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Preface to the 
Second Edition 


As you may know, the first edition of The Control Handbook was very well received. Many copies were 
sold and a gratifying number of people took the time to tell me that they found it useful. To the publisher, 
these are all reasons to do a second edition. To the editor of the first edition, these same facts are a modest 
disincentive. The risk that a second edition will not be as good as the first one is real and worrisome. I 
have tried very hard to insure that the second edition is at least as good as the first one was. I hope you 
agree that I have succeeded. 

I have made two major changes in the second edition. The first is that all the Applications chapters 
are new. It is simply a fact of life in engineering that once a problem is solved, people are no longer as 
interested in it as they were when it was unsolved. I have tried to find especially inspiring and exciting 
applications for this second edition. 

Secondly, it has become clear to me that organizing the Applications book by academic discipline is 
no longer sensible. Most control applications are interdisciplinary. For example, an automotive control 
system that involves sensors to convert mechanical signals into electrical ones, actuators that convert 
electrical signals into mechanical ones, several computers and a communication network to link sensors 
and actuators to the computers does not belong solely to any specific academic area. You will notice that 
the applications are now organized broadly by application areas, such as automotive and aerospace. 

One aspect of this new organization has created a minor and, I think, amusing problem. Several 
wonderful applications did not fit into my new taxonomy. I originally grouped them under the title 
Miscellaneous. Several authors objected to the slightly pejorative nature of the term “miscellaneous.” 
I agreed with them and, after some thinking, consulting with literate friends and with some of the 
library resources, I have renamed that section “Special Applications.” Regardless of the name, they are 
all interesting and important and I hope you will read those articles as well as the ones that did fit my 
organizational scheme. 

There has also been considerable progress in the areas covered in the Advanced Methods book. This 
is reflected in the roughly two dozen articles in this second edition that are completely new. Some of 
these are in two new sections, “Analysis and Design of Hybrid Systems” and “Networks and Networked 
Controls.” 

There have even been a few changes in the Fundamentals. Primarily, there is greater emphasis on 
sampling and discretization. This is because most control systems are now implemented digitally. 

I have enjoyed editing this second edition and learned a great deal while I was doing it. I hope that you 
will enjoy reading it and learn a great deal from doing so. 


William S. Levine 
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1-1 


1-2 Control System Advanced Methods 


Many of the problems considered in this chapter arise in the study of the “standard” linear model 


x(t) = Ax(t) + Bu(t), (1.1) 
y(t) = Cx(t) + Du(t). (1.2) 


Here, x(t) is an n-vector of states, u(t) is an m-vector of controls or inputs, and y(t) is a p-vector of 
outputs. The standard discrete-time analog of Equations 1.1 and 1.2 takes the form 


Xko1 = Axa + Bug, (1.3) 
Vk = Cxp + Dug. (1.4) 


Of course, considerably more elaborate models are also studied, including time-varying, stochastic, and 
nonlinear versions of the above, but these are not discussed in this chapter. In fact, the above linear models 
are usually derived from linearizations of nonlinear models regarding selected nominal points. 

The matrices considered here are, for the most part, assumed to have real coefficients and to be 
small (of order a few hundred or less) and dense, with no particular exploitable structure. Calculations 
for most problems in classical single-input, single-output control fall into this category. Large sparse 
matrices or matrices with special exploitable structures may significantly involve different concerns and 
methodologies than those discussed here. 

The systems, control, and estimation literature is replete with ad hoc algorithms to solve the compu- 
tational problems that arise in the various methodologies. Many of these algorithms work quite well on 
some problems (e.g., “small-order” matrices) but encounter numerical difficulties, often severe, when 
“pushed” (e.g., on larger order matrices). The reason for this is that little or no attention has been paid to 
the way algorithms perform in “finite arithmetic,” that is, on a finite word length digital computer. 

A simple example by Moler and Van Loan [14, p. 649]* illustrates a typical pitfall. Suppose it is desired 
to compute the matrix e“ in single precision arithmetic on a computer which gives six decimal places of 
precision in the fractional part of floating-point numbers. Consider the case 


—49 24 
ae Ee | 
and suppose the computation is attempted with the Taylor series formula 
A_Yo + ak 
gay ao (1.5) 


k=0 


This is easily coded and it is determined that the first 60 terms in the series suffice for the computation, in 
the sense that the terms for k > 60 of the order 10~7 no longer add anything significant to the sum. The 
resulting answer is 


—22.2588 —1.43277 
—61.4993 —3.47428 | ° 


Surprisingly, the true answer is (correctly rounded) 


—0.735759 0.551819 
—1.47152 1.10364 |" 


What happened here was that the intermediate terms in the series became very large before the factorial 
began to dominate. The 17th and 18th terms, for example, are of the order of 10” but of opposite signs so 


* The page number indicates the location of the appropriate reprint in [14]. 
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that the less significant parts of these numbers, while significant for the final answer, are “lost” because of 
the finiteness of the arithmetic. 

For this particular example, various fixes and remedies are available. However, in more realistic exam- 
ples, one seldom has the luxury of having the “true answer” available so that it is not always easy to simply 
inspect or test a computed solution and determine that it is erroneous. Mathematical analysis (truncation 
of the series, in the example above) alone is simply not sufficient when a problem is analyzed or solved in 
finite arithmetic (truncation of the arithmetic). Clearly, a great deal of care must be taken. 

The finiteness inherent in representing real or complex numbers as floating-point numbers on a digital 
computer manifests itself in two important ways: floating-point numbers have only finite precision 
and finite range. The degree of attention paid to these two considerations distinguishes many reliable 
algorithms from more unreliable counterparts. 

The development in systems, control, and estimation theory of stable, efficient, and reliable algorithms 
that respect the constraints of finite arithmetic began in the 1970s and still continues. Much of the research 
in numerical analysis has been directly applicable, but there are many computational issues in control 
(e.g., the presence of hard or structural zeros) where numerical analysis does not provide a ready answer 
or guide. A symbiotic relationship has developed, especially between numerical linear algebra and linear 
system and control theory, which is sure to provide a continuing source of challenging research areas. 

The abundance of numerically fragile algorithms is partly explained by the following observation: 


If an algorithm is amenable to “easy” manual calculation, it is probably a poor method if imple- 
mented in the finite floating-point arithmetic of a digital computer. 


For example, when confronted with finding the eigenvalues of a 2 x 2 matrix, most people would 
find the characteristic polynomial and solve the resulting quadratic equation. But when extrapolated 
as a general method for computing eigenvalues and implemented on a digital computer, this is a very 
poor procedure for reasons such as roundoff and overflow/underflow. The preferred method now would 
generally be the double Francis QR algorithm (see [17] for details) but few would attempt that manually, 
even for very small-order problems. 

Many algorithms, now considered fairly reliable in the context of finite arithmetic, are not amenable 
to manual calculations (e.g., various classes of orthogonal similarities). This is a kind of converse to 
the observation quoted above. Especially in linear system and control theory, we have been too easily 
tempted by the ready availability of closed-form solutions and numerically naive methods to implement 
those solutions. For example, in solving the initial value problem 


x(t) = Ax(t); x(0) = x0, (1.6) 


it is not at all clear that one should explicitly compute the intermediate quantity e’. Rather, it is the vector 
e'4xo that is desired, a quantity that may be computed by treating Equation 1.6 as a system of (possibly 
stiff) differential equations and using an implicit method for numerically integrating the differential 
equation. But such techniques are definitely not attractive for manual computation. 

The awareness of such numerical issues in the mathematics and engineering community has increased 
significantly in the last few decades. In fact, some of the background material well known to numer- 
ical analysts has already filtered down to undergraduate and graduate curricula in these disciplines. 
This awareness and education has affected system and control theory, especially linear system theory. 
A number of numerical analysts were attracted by the wealth of interesting numerical linear algebra 
problems in linear system theory. At the same time, several researchers in linear system theory turned to 
various methods and concepts from numerical linear algebra and attempted to modify them in develop- 
ing reliable algorithms and software for specific problems in linear system theory. This cross-fertilization 
has been greatly enhanced by the widespread use of software packages and by developments over the last 
couple of decades in numerical linear algebra. This process has already begun to have a significant impact 
on the future directions and development of system and control theory, and on applications, as evident 
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from the growth of computer-aided control system design (CACSD) as an intrinsic tool. Algorithms 
implemented as mathematical software are a critical “inner” component of a CACSD system. 

In the remainder of this chapter, we survey some results and trends in this interdisciplinary research 
area. We emphasize numerical aspects of the problems/algorithms, which is why we also spend time 
discussing appropriate numerical tools and techniques. We discuss a number of control and filtering 
problems that are of widespread interest in control. 

Before proceeding further, we list here some notations to be used: 


Bs the set of all n x m matrices with coefficients in the field F (F is generally R or C ) 
Al the transpose of A € R’*” 
Al the complex-conjugate transpose of A « C"*™ 
At the Moore-Penrose pseudoinverse of A 
|| Al the spectral norm of A (i-e., the matrix norm subordinate to the Euclidean vector 
norm: ||A|| = maxjx1,=1 ||Axll2) 
ay 0 
diag (a1,...,a,) the diagonal matrix 
0 an 
A(A) the set of eigenvalues i1,..., » (not necessarily distinct) of A € F”*” 
Ki(A) the ith eigenvalue of A 
x(A) the set of singular values 01,...,0m (not necessarily distinct) of A € F”*” 
0;(A) the ith singular value of A 


Finally, let us define a particular number to which we make frequent reference in the following. The 
machine epsilon or relative machine precision is defined, roughly speaking, as the smallest positive number 
€ that, when added to 1 on our computing machine, gives a number greater than 1. In other words, any 
machine representable number 8 less than € gets “ rounded off’ when (floating-point) added to 1 to give 
exactly 1 again as the rounded sum. The number ¢, of course, varies depending on the kind of computer 
being used and the precision of the computations (single precision, double precision, etc.). But the fact 
that such a positive number € exists is entirely a consequence of finite word length. 


1.2 Numerical Background 


In this section, we give a very brief discussion of two concepts fundamentally important in numerical 
analysis: numerical stability and conditioning. Although this material is standard in textbooks such as [8], 
it is presented here for completeness and because the two concepts are frequently confused in the systems, 
control, and estimation literature. 

Suppose we have a mathematically defined problem represented by f which acts on data d belonging to 
some set of data D, to produce a solution f(d) in a solution set S. These notions are kept deliberately vague 
for expository purposes. Given d € D, we desire to compute f(d). Suppose d* is some approximation to d. 
If f(d*) is “near” f(d), the problem is said to be well conditioned. If f(d*) may potentially differ greatly 
from f(d) even when d* is near d, the problem is said to be ill-conditioned. The concept of “near” can be 
made precise by introducing norms in the appropriate spaces. We can then define the condition of the 
problem f with respect to these norms as 


K[f (d)] = lim sup | (1.7) 


9 d(d,d*)=8 


ee] 
; : 
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where dj (-,-) are distance functions in the appropriate spaces. When k[f(d)] is infinite, the problem of 
determining f (d) from d is ill-posed (as opposed to well-posed). When k[f (d)] is finite and relatively large 
(or relatively small), the problem is said to be ill-conditioned (or well-conditioned). 

A simple example of an ill-conditioned problem is the following. Consider the n x n matrix 


0 1 0 0 
A= “all 
0 
‘ 1 
0 0 


with n eigenvalues at 0. Now, consider a small perturbation of the data (the n” elements of A) consisting 
of adding the number 2~” to the first element in the last (nth) row of A. This perturbed matrix then 
has n distinct eigenvalues 1,..., dn with X, = 1/2 exp(2kxj/n), where j := ./—1. Thus, we see that this 
small perturbation in the data has been magnified by a factor on the order of 2” resulting in a rather 
large perturbation in solving the problem of computing the eigenvalues of A. Further details and related 
examples can be found in [9,17]. 

Thus far, we have not mentioned how the problem f above (computing the eigenvalues of A in the 
example) was to be solved. Conditioning is a function solely of the problem itself. To solve a problem 
numerically, we must implement some numerical procedures or algorithms which we denote by f*. Thus, 
given d, f*(d) is the result of applying the algorithm to d (for simplicity, we assume d is “representable”; a 
more general definition can be given when some approximation d™ to d must be used). The algorithm f* 
is said to be numerically (backward) stable if, for all d € D, there exists d* € D near d so that f*(d) is near 
f(d*), (f (d*) = the exact solution of a nearby problem). If the problem is well-conditioned, then f(d*) 
is near f(d) so that f*(d) is near f(d) if f* is numerically stable. In other words, f* does not introduce 
any more sensitivity to perturbation than is inherent in the problem. Example 1.1 further illuminates this 
definition of stability which, on a first reading, can seem somewhat confusing. 

Of course, one cannot expect a stable algorithm to solve an ill-conditioned problem any more accurately 
than the data warrant, but an unstable algorithm can produce poor solutions even to well-conditioned 
problems. Example 1.2, illustrates this phenomenon. There are thus two separate factors to consider in 
determining the accuracy of a computed solution f*(d). First, ifthe algorithm is stable, f*(d) is near f(d*), 
for some d*, and second, if the problem is well conditioned, then, as above, f (d*) is near f (d). Thus, f*(d) 
is near f(d) and we have an “accurate” solution. 

Rounding errors can cause unstable algorithms to give disastrous results. However, it would be virtually 
impossible to account for every rounding error made at every arithmetic operation in a complex series of 
calculations. This would constitute a forward error analysis. The concept of backward error analysis based 
on the definition of numerical stability given above provides a more practical alternative. To illustrate this, 
let us consider the singular value decomposition (SVD) of an arbitrary m x n matrix A with coefficients 
in R or C [8] (see also Section 1.3.3), 


A=UxvV#. (1.8) 


Here U and V are m x mand n x n unitary matrices, respectively, and & is an m x n matrix of the form 
xu 0 
D= | an a x, = diag{o,...,0;} (1.9) 


with the singular values 0; positive and satisfying 0) > 02 --- > 0, > 0. The computation of this decom- 
position is, of course, subject to rounding errors. Denoting computed quantities by an overbar, for some 
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error matrix Ea, 


FZAS SUD (1.10) 


The computed decomposition thus corresponds exactly to a perturbed matrix A. When using the SVD 
algorithm available in the literature [8], this perturbation can be bounded by 


| Ea |< we |] A |], (1.11) 


where € is the machine precision and 1 is some quantity depending on the dimensions m and n, but 
reasonably close to 1 (see also [14, p. 74]). Thus, the backward error Ea induced by this algorithm has 
roughly the same norm as the input error E; resulting, for example, when reading the data A into the 
computer. Then, according to the definition of numerical stability given above, when a bound such as 
that in Equation 1.11 exists for the error induced by a numerical algorithm, the algorithm is said to be 
backward stable [17]. Note that backward stability does not guarantee any bounds on the errors in the 
result U, &, and V. In fact, this depends on how perturbations in the data (namely, E4 = A — A) affect 
the resulting decomposition (namely, Ey = U — U, Ey = Y—D, and Ey = V— V). This is commonly 
measured by the condition k[f(A)]. 

Backward stability is a property of an algorithm, and the condition is associated with a problem and 
the specific data for that problem. The errors in the result depend on the stability of the algorithm 
used and the condition of the problem solved. A good algorithm should, therefore, be backward stable 
because the size of the errors in the result is then mainly due to the condition of the problem, not to the 
algorithm. An unstable algorithm, on the other hand, may yield a large error even when the problem is 
well conditioned. 

Bounds of the type Equation 1.11 are obtained by an error analysis of the algorithm used, and the 
condition of the problem is obtained by a sensitivity analysis; for example, see [9,17]. 

We close this section with two simple examples to illustrate some of the concepts introduced. 


Example 1.1: 


Let x and y be two floating-point computer numbers and let fl(x * y) denote the result of multiplying 
them in floating-point computer arithmetic. In general, the product x « y requires more precision to 
be represented exactly than was used to represent x or y. But for most computers 


f(x xy) =x xy(1+8), (1.12) 


where |8| < € (= relative machine precision). In other words, fl(x * y) is x * y correct to within a unit in 
the last place. Another way to write Equation 1.12 is as follows: 


f(x xy) = x(1+8)'/2 x y(1 4.8)'/2, (1.13) 


where |8| < «. This can be interpreted as follows: the computed result fl(x * y) is the exact prod- 
uct of the two slightly perturbed numbers x(1 + §)'/2 and y(1+ §)1/2, The slightly perturbed data 
(not unique) may not even be representable as floating-point numbers. The representation of 
Equation 1.13 is simply a way of accounting for the roundoff incurred in the algorithm by an ini- 
tial (small) perturbation in the data. 
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Example 1.2: 


Gaussian elimination with no pivoting for solving the linear system of equations 
Ax=b (1.14) 


is known to be numerically unstable; see for example [8] and Section 1.3. The following data 
illustrate this phenomenon. Let 


4 — {0.0001 1.000], _ [1.000 
~}1.000 -1.000]’ ~~ |0.000]" 


All computations are carried out in four-significant-figure decimal arithmetic. The “true answer” 


x=A7'bis 
0.9999 
0.9999 | ° 


Using row 1 as the “pivot row” (i.e., subtracting 10,000 x row 1 from row 2) we arrive at the equivalent 


triangular system 
0.0001 1.000 xy} 1.000 
0 —1.000 x 104] | x2 | ~ | —1.000 x 104 | ° 


The coefficient multiplying xz in the second equation should be —10, 001, but because of roundoff, 
becomes — 10,000. Thus, we compute x2 = 1.000 (a good approximation), but back substitution in 
the equation 

0.0001x; = 1.000 — fl(1.000 « 1.000) 


yields x; = 0.000. This extremely bad approximation to x, is the result of numerical instability. The 
problem, it can be shown, is quite well conditioned. 


1.3 Fundamental Problems in Numerical Linear Algebra 


In this section, we give a brief overview of some of the fundamental problems in numerical linear algebra 
that serve as building blocks or “tools” for the solution of problems in systems, control, and estimation. 


1.3.1 Linear Algebraic Equations and Linear Least-Squares Problems 


Probably the most fundamental problem in numerical computing is the calculation of a vector x which 
satisfies the linear system 
Ax = b, (1.15) 


where A € R"*"(or C”*") and has rank n. A great deal is now known about solving Equation 1.15 in finite 
arithmetic both for the general case and for a large number of special situations, for example, see [8,9]. 

The most commonly used algorithm for solving Equation 1.15 with general A and small n (say n < 1000) 
is Gaussian elimination with some sort of pivoting strategy, usually “partial pivoting.” This amounts to 
factoring some permutation of the rows of A into the product of a unit lower triangular matrix L and an 
upper triangular matrix U. The algorithm is effectively stable, that is, it can be proved that the computed 
solution is near the exact solution of the system 


(A+E)x=b (1.16) 


with |e;| < (1) y B¢, where $(n) is a modest function of n depending on details of the arithmetic used, y 
is a “growth factor” (which is a function of the pivoting strategy and is usually—but not always—small), 
B behaves essentially like ||A||, and € is the machine precision. In other words, except for moderately 
pathological situations, E is “small”—on the order of € || Al]. 
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The following question then arises. If, because of rounding errors, we are effectively solving Equa- 
tion 1.16 rather than Equation 1.15, what is the relationship between (A + E)~'b and A~'b? To answer 
this question, we need some elementary perturbation theory and this is where the notion of condition 
number arises. A condition number for Equation 1.15 is given by 


k(A) : = ||Al] ||A7 I. (1.17) 


Simple perturbation results can show that perturbation in A and/or b can be magnified by as much as 
k(A) in the computed solution. Estimating k(A) (since, of course, A7! is unknown) is thus a crucial 
aspect of assessing solutions of Equation 1.15 and the particular estimating procedure used is usually the 
principal difference between competing linear equation software packages. One of the more sophisticated 
and reliable condition estimators presently available is implemented in LINPACK [5] and its successor 
LAPACK [2]. LINPACK and LAPACK also feature many codes for solving Equation 1.14 in case A has 
certain special structures (e.g., banded, symmetric, or positive definite). 

Another important class of linear algebra problems, and one for which codes are available in LINPACK 
and LAPACK, is the linear least-squares problem 


min ||Ax — Dll, (1.18) 


where A € R™*” and has rank k, with (in the simplest case) k = n < m, for example, see [8]. The solution 
of Equation 1.18 can be written formally as x = At b. The method of choice is generally based on the QR 
factorization of A (for simplicity, let rank(A) = n) 


A= QR, (1.19) 


where R € R"*" is upper triangular and Q € R”*" has orthonormal columns, that is, Q7Q =I. With 
special care and analysis, the case k < ncan also be handled similarly. The factorization is effected through 
a sequence of Householder transformations H; applied to A. Each Hj is symmetric and orthogonal and 
of the form I — 2uu? /u? u, where u € R™ is specially chosen so that zeros are introduced at appropriate 
places in A when it is premultiplied by Hj. After n such transformations, 


AyHy-1... A= | , 


from which the factorization Equation 1.19 follows. Defining c and d by 


c 
|; | = Ay,Ay-1 oe: . Hb, 


where c € R", it is easily shown that the least-squares solution x of Equation 1.18 is given by the solution 
of the linear system of equations 


Rx=c. (1.20) 


The above algorithm is numerically stable and, again, a well-developed perturbation theory exists from 
which condition numbers can be obtained, this time in terms of 


k(A) : = ||Al] ||AT I. 


Least-squares perturbation theory is fairly straightforward when rank(A) = n, but is considerably more 
complicated when A is rank deficient. The reason for this is that, although the inverse is a continuous 
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function of the data (i.e., the inverse is a continuous function in a neighborhood of a nonsingular matrix), 
the pseudoinverse is discontinuous. For example, consider 


and perturbations 


with 8 being small. Then 


1 8 
(A+E})* =| 14+82 148], 
0 0 
which is close to At but 
1 0 
(A+ Ey) = ae 
8 


which gets arbitrarily far from A* as 8 is decreased toward 0. 

In lieu of Householder transformations, Givens transformations (elementary rotations or reflections) 
may also be used to solve the linear least-squares problem [8]. Givens transformations have received 
considerable attention for solving linear least-squares problems and systems of linear equations in a 
parallel computing environment. The capability of introducing zero elements selectively and the need for 
only local interprocessor communication make the technique ideal for “parallelization.” 


1.3.2 Eigenvalue and Generalized Eigenvalue Problems 


In the algebraic eigenvalue/eigenvector problem for A € R”*”, one seeks nonzero solutions x € C” and 
» € C, which satisfy 
Ax = hx. (1.21) 


The classic reference on the numerical aspects of this problem is Wilkinson [17]. A briefer textbook 
introduction is given in [8]. 

Quality mathematical software for eigenvalues and eigenvectors is available; the EISPACK [7,15] 
collection of subroutines represents a pivotal point in the history of mathematical software. The suc- 
cessor to EISPACK (and LINPACK) is LAPACK [2], in which the algorithms and software have been 
restructured to provide high efficiency on vector processors, high-performance workstations, and shared 
memory multiprocessors. 

The most common algorithm now used to solve Equation 1.21 for general A is the QR algorithm 
of Francis [17]. A shifting procedure enhances convergence and the usual implementation is called the 
double-Francis-QR algorithm. Before the QR process is applied, A is initially reduced to upper Hessenberg 
form Ay (aj = 0 if i—j = 2). This is accomplished by a finite sequence of similarities of the Householder 
form discussed above. The QR process then yields a sequence of matrices orthogonally similar to A and 
converging (in some sense) to a so-called quasi-upper triangular matrix S also called the real Schur form 
(RSF) of A. The matrix S is block upper triangular with 1 x 1 diagonal blocks corresponding to real 
eigenvalues of A and 2 x 2 diagonal blocks corresponding to complex-conjugate pairs of eigenvalues. The 
quasi-upper triangular form permits all arithmetic to be real rather than complex as would be necessary 
for convergence to an upper triangular matrix. The orthogonal transformations from both the Hessenberg 
reduction and the QR process may be accumulated in a single orthogonal transformation U so that 


UTAU=R (1.22) 


compactly represents the entire algorithm. An analogous process can be applied in the case of symmetric 
A, and considerable simplifications and specializations result. 
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Closely related to the QR algorithm is the QZ algorithm for the generalized eigenvalue problem 
Ax = Mx, (1.23) 


where A, M € R”*”. Again, a Hessenberg-like reduction, followed by an iterative process, is implemented 
with orthogonal transformations to reduce Equation 1.23 to the form 


QAZy = QMZy, (1.24) 


where QAZ is quasi-upper triangular and QMZ is upper triangular. For a review and references to results 
on stability, conditioning, and software related to Equation 1.23 and the QZ algorithm, see [8]. The 
generalized eigenvalue problem is both theoretically and numerically more difficult to handle than the 
ordinary eigenvalue problem, but it finds numerous applications in control and system theory [14, p. 109]. 


1.3.3 The Singular Value Decomposition and Some Applications 


One of the basic and most important tools of modern numerical analysis, especially numerical linear 
algebra, is the SVD. Here we make a few comments about its properties and computation as well as its 
significance in various numerical problems. 

Singular values and the SVD have a long history, especially in statistics and numerical linear algebra. 
These ideas have found applications in the control and signal processing literature, although their use 
there has been overstated somewhat in certain applications. For a survey of the SVD, its history, numerical 
details, and some applications in systems and control theory, see [14, p. 74]. 

The fundamental result was stated in Section 1.2 (for the complex case). The result for the real case is 
similar and is stated below. 


Theorem 1.1: 


Let A € R™*" with rank(A) = r. Then there exist orthogonal matrices U € R™*™ and V € R"*" so that 


A=UxV', (1.25) 
where 
x, 0 
=|) | 
and &, = diag {o1,...,0,} witho, >--->0, >0. 


The proof of Theorem 1.1 is straightforward and can be found, for example, in [8]. Geometrically, the 
theorem says that bases can be found (separately) in the domain and codomain spaces of a linear map 
with respect to which the matrix representation of the linear map is diagonal. The numbers 0},...,0;, 
together with 0,4; =0,...,0, = 0, are called the singular values of A, and they are the positive square 
roots of the eigenvalues of A'A. The columns {uz,k = 1,...,m} of U are called the left singular vectors 
of A (the orthonormal eigenvectors of AA"), while the columns {v;, k = 1,...,”} of V are called the right 
singular vectors of A (the orthonormal eigenvectors of A? A). The matrix A can then be written (as a 
dyadic expansion) also in terms of the singular vectors as follows: 


i 
A= ye OLUKY, « 
k=1 


The matrix A’ has m singular values, the positive square roots of the eigenvalues of AA’. The r 
[= rank(A)] nonzero singular values of A and A! are, of course, the same. The choice of A’ A rather than 
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AA? in the definition of singular values is arbitrary. Only the nonzero singular values are usually of any 
real interest and their number, given the SVD, is the rank of the matrix. Naturally, the question of how to 
distinguish nonzero from zero singular values in the presence of rounding error is a nontrivial task. 

It is not generally advisable to compute the singular values of A by first finding the eigenvalues of A? A, 
tempting as that is. Consider the following example, where jt is a real number with || < ./e (so that 
fl +7) = 1, where fl(-) denotes floating-point computation). Let 


A= 


oS = 
= Or 


Then 
T,,_{1 1 
fla ay=|t i}: 


So we compute 6) = J/2, 62 =0 leading to the (erroneous) conclusion that the rank of A is 1. Of course, 
if we could compute in infinite precision, we would find 


T 42 1+? 1 
Atas| 1 1+? 


with 0) = 2+ 2,02 = |w| and thus rank(A) = 2. The point is that by working with A'A we have 
unnecessarily introduced 1? into the computations. The above example illustrates a potential pitfall in 
attempting to form and solve the normal equations in a linear least-squares problem and is at the heart 
of what makes square root filtering so attractive numerically. Very simplistically, square root filtering 
involves working directly on an “A-matrix,” for example, updating it, as opposed to updating an “A? A- 
matrix.” 

Square root filtering is usually implemented with the QR factorization (or some closely related algo- 
rithm) as described previously rather than SVD. Moreover, critical information may be lost irrecoverably 
by simply forming A’ A. 

Returning now to the SVD, two features of this matrix factorization make it attractive in finite arith- 
metic: first, it can be computed in a numerically stable way, and second, singular values are well con- 
ditioned. Specifically, there is an efficient and numerically stable algorithm by Golub and Reinsch [8] 
which works directly on A to give the SVD. This algorithm has two phases. In the first phase, it computes 
orthogonal matrices U, and V, so that B= Uj, TAV, is in bidiagonal form, that is, only the elements on 
its diagonal and first superdiagonal are nonzero. In the second phase, the algorithm uses an iterative 
procedure to compute orthogonal matrices U2 and V2 so that U2? BV? is diagonal and nonnegative. The 
SVD defined in Equation 1.25 is then © = U' BV, where U = U, Up and V = V,V>. The computed U 
and V are orthogonal approximately to the working precision, and the computed singular values are the 
exact o;’s for A+ E, where ||E||/||A|| is a modest multiple of €. Fairly sophisticated implementations of 
this algorithm can be found in [5,7]. The well-conditioned nature of the singular values follows from the 
fact that if A is perturbed to A + E, then it can be proved that 


|oi(A + E) — 0;(A)|| < IIE. 


Thus, the singular values are computed with small absolute error although the relative error of sufficiently 
small singular values is not guaranteed to be small. 

It is now acknowledged that the singular value decomposition is the most generally reliable method of 
determining rank numerically (see [14, p. 589] for a more elaborate discussion). However, it is consider- 
ably more expensive to compute than, for example, the QR factorization which, with column pivoting [5], 
can usually give equivalent information with less computation. Thus, while the SVD is a useful theoretical 
tool, its use for actual computations should be weighed carefully against other approaches. 
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The problem of numerical determination of rank is now well understood. The essential idea is to try 
to determine a “gap” between “zero” and the “smallest nonzero singular value” of a matrix A. Since the 
computed values are exact for a matrix near A, it makes sense to consider the ranks of all matrices in 
some 8-ball (with respect to the spectral norm || - ||, say) around A. The choice of 8 may also be based 
on measurement errors incurred in estimating the coefficients of A, or the coefficients may be uncertain 
because of rounding errors incurred in a previous computation. However, even with SVD, numerical 
determination of rank in finite arithmetic is a difficult problem. 

That other methods of rank determination are potentially unreliable is demonstrated by the following 
example. Consider the Ostrowski matrix A € R”’*” whose diagonal elements are all —1, whose upper 
triangle elements are all +1, and whose lower triangle elements are all 0. This matrix is clearly of rank n, 
that is, is invertible. It has a good “solid” upper triangular shape. All of its eigenvalues (—1) are well away 
from zero. Its determinant (—1)” is definitely not close to zero. But this matrix is, in fact, very nearly 
singular and becomes more nearly so as n increases. Note, for example, that 


0 te Ee : 1 2701 0 
. . . . 2-1 2701 0 
= => (n > +00). 
a-m+l 2-11 0 
: fe SEL 


Moreover, adding 2~"*! to every element in the first column of A gives an exactly singular matrix. 
Arriving at such a matrix by, say, Gaussian elimination would give no hint as to the near singularity. 
However, it is easy to check that o,,(A) behaves as 2~"*!. A corollary for control theory is that eigenvalues 
do not necessarily give a reliable measure of “stability margin.” It is useful to note that in this example 
of an invertible matrix, the crucial quantity, o,(A), which measures nearness to singularity, is simply 
1/||A7!|], and the result is familiar from standard operator theory. There is nothing intrinsic about 
singular values in this example and, in fact, ||A~!|| might be more cheaply computed or estimated in 
other matrix norms. 

Because rank determination, in the presence of rounding error, is a nontrivial problem, the same diffi- 
culties naturally arise in any problem equivalent to, or involving, rank determination, such as determining 
the independence of vectors, finding the dimensions of certain subspaces, etc. Such problems arise as basic 
calculations throughout systems, control, and estimation theory. Selected applications are discussed in 
more detail in [14, p. 74] and in [1,4,10]. 

Finally, let us close this section with a brief example illustrating a totally inappropriate use of SVD. The 
rank condition 

rank [B, AB,...,A”- 1B] =n (1.26) 


for the controllability of Equation 1.1 is too well known. Suppose 


with || < /e. Then 
1 1 
B, AB] = . 
ua.api=| i] 


and now even applying SVD, the erroneous conclusion of uncontrollability is reached. Again the problem 
is in just forming AB; not even SVD can come to the rescue after that numerical faux pas. 
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1.4 Applications to Systems and Control 


A reasonable approach to developing numerically reliable algorithms for computational problems in 
linear system theory would be to reformulate the problems as concatenations of subproblems for which 
numerically stable algorithms are available. Unfortunately, one cannot ensure that the stability of algo- 
rithms for the subproblems results in the stability of the overall algorithm. This requires separate analysis 
that may rely on the sensitivity or condition of the subproblems. In the next section, we show that delicate 
(ie., badly conditioned) subproblems should be avoided whenever possible; a few examples are given 
where a possibly badly conditioned step is circumvented by carefully modifying or completing existing 
algorithms; see, for example, [14, p. 109]. 

A second difficulty is the ill-posedness of some of the problems occurring in linear system theory. Two 
approaches can be adopted. One can develop an acceptable perturbation theory for such problems, using 
a concept such as restricted condition, which is, the condition under perturbations for which a certain 
property holds, for example, fixed rank [14, p. 109]. One then looks for restricting assumptions that make 
the problem well posed. Another approach is to delay any such restricting choices to the end and leave 
it to the user to decide which choice to make by looking at the results. The algorithm then provides 
quantitative measures that help the user make this choice; see, for example, [14, p. 171, 529]. By this 
approach, one may avoid artificial restrictions of the first approach that sometimes do not respect the 
practical significance of the problem. 

A third possible pitfall is that many users almost always prefer fast algorithms to slower ones. However, 
slower algorithms are often more reliable. 

In the subsections that follow, we survey a representative selection of numerical linear algebra problems 
arising in linear systems, control, and estimation theory, which have been examined with some of the 
techniques described in the preceding sections. Many of these topics are discussed briefly in survey papers 
such as [11] and [16] and in considerably more detail in the papers included or referenced in [14] and 
in [1,4,10]. Some of the scalar algorithms discussed here do not extend trivially to the matrix case. When 
they do, we mention only the matrix case. Moreover, we discuss only the numerical aspects here; for the 
system-theoretical background, we refer the reader to the control and systems literature. 


1.4.1 Some Typical Techniques 


Most of the reliable techniques in numerical linear algebra are based on the use of orthogonal trans- 
formations. Typical examples of this are the QR decomposition for least-squares problems, the Schur 
decomposition for eigenvalue and generalized eigenvalue problems, and the SVD for rank determina- 
tions and generalized inverses. Orthogonal transformations also appear in most of the reliable linear 
algebra techniques for control theory. This is partly due to the direct application of existing linear algebra 
decompositions to problems in control. Obvious examples of this are the Schur approach for solving alge- 
braic Riccati equations, both continuous- and discrete-time [14, p. 529, 562, 573], for solving Lyapunov 
equations [14, p. 430] and for performing pole placement [14, p. 415]. New orthogonal decompositions 
have also been introduced that rely heavily on the same principles but were specifically developed for 
problems encountered in control. Orthogonal state-space transformations on a system {A, B, C} result in 
a new state-space representation {U“ AU, UB, CU}, where U performs some kind of decomposition on 
the matrices A, B, and C. These special forms, termed “condensed forms,” include 


¢ The state Schur form [14, p. 415] 

« The state Hessenberg form [14, p. 287] 

¢ The observer Hessenberg form [14, p. 289, 392] 

¢ The controller Hessenberg form [14, p. 128, 357]. 


Staircase forms or block Hessenberg forms are other variants of these condensed forms that have 
proven useful in dealing with MIMO systems [14, p. 109, 186, 195]. 
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There are two main reasons for using these orthogonal state-space transformations: 


¢ The numerical sensitivity of the control problem being solved is not affected by these transforma- 
tions because sensitivity is measured by norms or angles of certain spaces and these are unaltered 
by orthogonal transformations. 

¢ Orthogonal transformations have minimum condition number, essential in proving bounded error 
propagation and establishing numerical stability of the algorithm that uses such transformations. 


More details on this are given in [14, p. 128] and in subsequent sections where some of these condensed 
forms are used for particular applications. 


1.4.2 Transfer Functions, Poles, and Zeros 


In this section, we discuss important structural properties of linear systems and the numerical techniques 
available for determining them. The transfer function R() of a linear system is given by a polynomial 
representation VONT~!(A)U(CA) + WOK) or by a state-space model C(AI — A)~!B+D. The results in 
this subsection hold for both the discrete-time case (where ) stands for the shift operator z) and the 
continuous-time case (where i stands for the differentiation operator D). 


1.4.2.1 The Polynomial Approach 


One is interested in a number of structural properties of the transfer function R(), such as poles, 
transmission zeros, decoupling zeros, etc. In the scalar case, where {T(d), U(A), V(A), W(A)} are scalar 
polynomials, all of this can be found with a greatest common divisor (GCD) extraction routine and a 
rootfinder, for which reliable methods exist. In the matrix case, the problem becomes much more complex 
and the basic method for GCD extraction, the Euclidean algorithm, becomes unstable (see [14, p. 109]). 
Moreover, other structural elements (null spaces, etc.) come into the picture, making the polynomial 
approach less attractive than the state-space approach [14, p. 109, and references therein]. 


1.4.2.2 The State-Space Approach 


The structural properties of interest are poles and zeros of R(X), decoupling zeros, controllable and 
unobservable subspaces, supremal (A, B)-invariant and controllability subspaces, factorizability of R(\), 
left and right null spaces of R(X), etc. These concepts are fundamental in several design problems and 
have received considerable attention over the last few decades; see, for example, [14, p. 74, 109, 174, 
186, 529]. In [14, p. 109], it is shown that all the concepts mentioned above can be considered gener- 
alized eigenstructure problems and that they can be computed via the Kronecker canonical form of the 
pencils 


PA] [MI — A |B] 
at MI-A|B (1.27) 
=6 SCF iD 


or from other pencils derived from these. Backward stable software is also available for computing the 
Kronecker structure of an arbitrary pencil. A remaining problem here is that determining several of the 
structural properties listed above may be ill-posed in some cases in which one has to develop the notion 
of restricted condition (see [14, p. 109]). A completely different approach is to reformulate the problem as 
an approximation or optimization problem for which quantitative measures are derived, leaving the final 
choice to the user. Results in this vein are obtained for controllability, observability [14, p. 171, 186, 195] 
(almost) (A, B)-invariant, and controllability subspaces. 
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1.4.3 Controllability and Other “Abilities” 


The various “abilities” such as controllability, observability, reachability, reconstructibility, stabilizability, 
and detectability are basic to the study of linear control and system theory. These concepts can also be 
viewed in terms of decoupling zeros, controllable and unobservable subspaces, controllability subspaces, 
etc. mentioned in the previous section. Our remarks here are confined, but not limited, to the notion of 
controllability. 

A large number of algebraic and dynamic characterizations of controllability have been given; see [11] 
for a sample. But each one of these has difficulties when implemented in finite arithmetic. For a survey 
of this topic and numerous examples, see [14, p. 186]. Part of the difficulty in dealing with controllability 
numerically lies in the intimate relationship with the invariant subspace problem [14, p. 589]. The 
controllable subspace associated with Equation 1.1 is the smallest A-invariant subspace (subspace spanned 
by eigenvectors or principal vectors) containing the range of B. Since the A-invariant subspaces can be 
extremely sensitive to perturbation, it follows that, so too, is the controllable subspace. Similar remarks 
apply to the computation of the so-called controllability indices. The example discussed in the third 
paragraph of Section 1.2 dramatically illustrates these remarks. The matrix A has but one eigenvector 
(associated with 0), whereas the slightly perturbed A has n eigenvectors associated with the n distinct 
eigenvalues. 

Attempts have been made to provide numerically stable algorithms for the pole placement problem 
discussed in a later section. It suffices to mention here that the problem of pole placement by state feedback 
is closely related to controllability. Work on developing numerically stable algorithms for pole placement 
is based on the reduction of A to a Hessenberg form; see, for example, [14, p. 357, 371, 380]. In the 
single-input case, a good approach is the controller Hessenberg form mentioned above where the state 
matrix A is upper Hessenberg and the input vector B is a multiple of (1,0, ...,0)". The pair (A, B) is then 
controllable if, and only if, all (n — 1) subdiagonal elements of A are nonzero. Ifa subdiagonal element is 0, 
the system is uncontrollable, and a basis for the uncontrollable subspace is easily constructed. The transfer 
function gain or first nonzero Markov parameter is also easily constructed from this “canonical form.” 
In fact, the numerically more robust system Hessenberg form, playing an ever-increasing role in system 
theory, is replacing the numerically more fragile special case of the companion or rational canonical or 
Luenberger canonical form. 

A more important aspect of controllability is a topological notion such as “near uncontrollability.” But 
there are numerical difficulties here also, and we refer to Parts 3 and 4 of [14] for further details. Related to 
this is an interesting system-theoretic concept called “balancing” discussed in Moore’s paper [14, p. 171]. 
The computation of “balancing transformations” is discussed in [14, p. 642]. 

There are at least two distinct notions of near uncontrollability [11] in the parametric sense and in 
the energy sense. In the parametric sense, a controllable pair (A, B) is said to be near uncontrollable 
if the parameters of (A, B) need be perturbed by only a relatively small amount for (A, B) to become 
uncontrollable. In the energy sense, a controllable pair is near-uncontrollable if large amounts of control 
energy (' u"u) are required for a state transfer. The pair 


is very near-uncontrollable in the energy sense but not as badly as in the parametric sense. Of course, 
both measures are coordinate dependent and “balancing” is one attempt to remove this coordinate bias. 
The pair (A, B) above is in “controllable canonical form.” It is now known that matrices in this form 
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(specifically, the A matrix in rational canonical form) almost always exhibit poor numerical behavior and 
are “close to” uncontrollable (unstable, etc.) as the size n increases. For details, see [14, p. 59]. 


1.4.4 Computation of Objects Arising in the Geometric Theory 
of Linear Multivariable Control 


A great many numerical problems arise in the geometric approach to control of systems modeled as 
Equations 1.1 and 1.2. Some of these are discussed in the paper by Klema and Laub [14, p. 74]. The power 
of the geometric approach derives in large part from the fact that it is independent of specific coordinate 
systems or matrix representations. Numerical issues are a separate concern. 

A very thorough numerical treatment of numerical problems in linear system theory has been given 
by Van Dooren [14, p. 109]. This work has applications for most calculations with linear state-space 
models. For example, one by-product is an extremely reliable algorithm (similar to an orthogonal version 
of Silverman’s structure algorithm) for the computation of multivariable system zeros [14, p. 271]. This 
method involves a generalized eigenvalue problem (the Rosenbrock pencil), but the “infinite zeros” are 
first removed by deflating the given matrix pencil. 


1.4.5 Frequency Response Calculations 


Many properties of a linear system such as Equations 1.1 and 1.2 are known in terms of its frequency 
response matrix 


G(jw) : = C(jol —A)"1B+D; (w>0) (1.28) 


(or G(e®); 6 € [0,27] for Equations 1.3 and 1.4). In fact, various norms of the return difference matrix 
I+ G(jw) and related quantities have been investigated in control and system theory to providing robust 
linear systems with respect to stability, noise response, disturbance attenuation, sensitivity, etc. 

Thus it is important to compute G(jw) efficiently, given A, B, and C for a (possibly) large number 
of values of w (for convenience we take D to be 0, because if it is nonzero it is trivial to add to G). An 
efficient and generally applicable algorithm for this problem is presented in [14, p. 287]. Rather than 
solving the linear equation (j@I — A)X = B with dense unstructured A, which would require O(n?) oper- 
ations for each successive value of w, the new method initially reduces A to upper Hessenberg form 
H. The orthogonal state-space coordinate transformations used to obtain the Hessenberg form of A 
are incorporated into B and C giving B and C. As w varies, the coefficient matrix in the linear equa- 
tion (j@I — H)X = B remains in upper Hessenberg form. The advantage is that X can now be found 
in O(n?) operations rather than O(n?) as before, a substantial saving. Moreover, the method is numeri- 
cally very stable (via either LU or QR factorization) and has the advantage of being independent of the 
eigenstructure (possibly ill-conditioned) of A. Another efficient and reliable algorithm for frequency 
response computation [14, p. 289] uses the observer Hessenberg form mentioned in Section 1.4.1 
together with a determinant identity and a property of the LU decomposition of a Hessenberg matrix. 

The methods above can also be extended to state-space models in implicit form, that is, where Equa- 
tion 1.1 is replaced by 


Ex = Ax + Bu. (1.29) 


Then Equation 1.28 is replaced with 


G(jw) = C(jwE — A)~'B+D, (1.30) 


and the initial triangular/Hessenberg reduction [8] can be employed again to reduce the problem to 
updating the diagonal of a Hessenberg matrix and consequently an O(n”) problem. 

An improvement for the frequency response evaluation problem is using matrix interpolation methods 
to achieve even greater computational efficiency. 
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1.4.6 Numerical Solution of Linear Ordinary Differential Equations 
and Matrix Exponentials 


The “simulation” or numerical solution of linear systems of ordinary differential equations (ODEs) of 
the form 
x(t) = Ax(t)+f(t), (0) =xo0, (1.31) 


is a standard problem that arises in finding the time response of a system in state-space form. However, 
there is still debate as to the most effective numerical algorithm, particularly when A is defective (i.e., 
when A is n x n and has fewer than n linearly independent eigenvectors) or nearly defective. The most 
common approach involves computing the matrix exponential e, because the solution of Equation 1.31 
can be written simply as 
t 
x(t) = e!Axy + / el “JAF (5) ds, 
0 

A delightful survey of computational techniques for matrix exponentials is given in [14, p. 649]. Nine- 
teen “dubious” ways are explored (there are many more ways not discussed) but no clearly superior 
algorithm is singled out. Methods based on Padé approximation or reduction of A to RSF are generally 
attractive while methods based on Taylor series or the characteristic polynomial of A are generally found 
unattractive. An interesting open problem is the design of a special algorithm for the matrix exponential 
when the matrix is known a priori to be stable (A(A) in the left half of the complex plane). 

The reason for the adjective “dubious” in the title of [14, p. 649] is that in many (maybe even most) 
circumstances, it is better to treat Equation 1.31 as a system of differential equations, typically stiff, and to 
apply various ODE techniques, specially tailored to the linear case. ODE techniques are preferred when 
A is large and sparse for, in general, e' is unmanageably large and dense. 


1.4.7 Lyapunov, Sylvester, and Riccati Equations 


Certain matrix equations arise naturally in linear control and system theory. Among those frequently 
encountered in the analysis and design of continuous-time systems are the Lyapunov equation 


AX +XA'+Q=0, (1.32) 


and the Sylvester equation 
AX+XF+Q=0. (1.33) 


The appropriate discrete-time analogs are 
AXAT—~X+Q=0 (1.34) 


and 
AXF-X+Q=0. (1.35) 


Various hypotheses are posed for the coefficient matrices A, F, and Q to ensure certain properties of 
the solution X. 

The literature in control and system theory on these equations is voluminous, but most of it is ad hoc, at 
best, from a numerical point of view, with little attention to questions of numerical stability, conditioning, 
machine implementation, and the like. 

For the Lyapunov equation, the best overall algorithm in terms of efficiency, accuracy, reliability, 
availability, and ease of use is that of Bartels and Stewart [14, p. 430]. The basic idea is to reduce A to 
quasi-upper triangular form (or RSF) and to perform a back substitution for the elements of X. 

An attractive algorithm for solving Lyapunov equations has been proposed by Hammarling [14, p. 500]. 
This algorithm is a variant of the Bartels—Stewart algorithm but instead solves directly for the Cholesky 
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factor Y of X: Y'Y =X and Y is upper triangular. Clearly, given Y, X is easily recovered if necessary. 
But in many applications, for example, [14, p. 642] only the Cholesky factor is required. 

For the Lyapunov equation, when A is stable, the solutions of the equations above are also equal to 
the reachability and observability Grammians P,(T) and P,(T), respectively, for T = +00 for the system 
{A, B, C}: 


T : ue 
P,(T) = / e'ABB! ef" dt; P,(T) = / el CT CelAdt 
0 0 


T T (1.36) 
PT)= >" Atep (AS BT) = > (ADC aa. 
k=0 k=0 

These can be used along with some additional transformations (see [14, p. 171, 642]) to compute the 
so-called balanced realizations {A, B, C}. For these realizations, both P, and P, are equal and diagonal. 
These realizations have some nice sensitivity properties with respect to poles, zeros, truncation errors in 
digital filter implementations, etc. [14, p. 171]. They are, therefore, recommended whenever the choice 
of a realization is left to the user. When A is not stable, one can still use the finite range Grammians 
Equation 1.36, for T < +00, for balancing [14, p. 171]. A reliable method for computing integrals and 
sums of the type Equation 1.36 can be found in [14, p. 681]. It is also shown in [14, p. 171] that the 
reachable subspace and the unobservable subspace are the image and the kernel of P,(T) and P,(T), 
respectively. From these relationships, sensitivity properties of the spaces under perturbations of P,(T) 
and P,(T) can be derived. 

For the Sylvester equation, the Bartels-Stewart algorithm reduces both A and F to RSF and then a back 
substitution is done. It has been demonstrated in [14, p. 495] that some improvement in this procedure 
is possible by only reducing the larger of A and F to upper Hessenberg form. The stability of this method 
has been analyzed in [14, p. 495]. Although only weak stability is obtained, this is satisfactory in most 
cases. 

Algorithms are also available in the numerical linear algebra literature for the more general Sylvester 
equation 

A\XF,' +A.XF,7+Q=0 


and its symmetric Lyapunov counterpart 
AXF’ + FXA'+Q=0. 


Questions remain about estimating the condition of Lyapunov and Sylvester equations efficiently and 
reliably in terms of the coefficient matrices. A deeper analysis of the Lyapunov and Sylvester equations is 
probably a prerequisite to at least a better understanding of condition of the Riccati equation for which, 
again, there is considerable theoretical literature but not as much known from a purely numerical point 
of view. The symmetric n x n algebraic Riccati equation takes the form 


Q+AX+XA™ —XGX=0 (1.37) 
for continuous-time systems and 
ATXA—X—A'XG\(G.+ G] XG)" 'G{XA+Q=0 (1.38) 


for discrete-time systems. These equations appear in several design/analysis problems, such as optimal 
control, optimal filtering, spectral factorization, for example, see the papers in Part 7 of [14] and references 
therein. Again, appropriate assumptions are made on the coefficient matrices to guarantee the existence 
and/or uniqueness of certain kinds of solutions X. Nonsymmetric Riccati equations of the form 


Q+A,X + XA, — XGX =0 (1.39) 


for the continuous-time case (along with an analog for the discrete-time case) are also studied and can be 
solved numerically by the techniques discussed below. 
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Several algorithms have been proposed based on different approaches. One of the more reliable general- 
purpose methods for solving Riccati equations is the Schur method [14, p. 529]. For the case of Equa- 
tion 1.37, for example, this method is based on reducing the associated 2n x 2n Hamiltonian matrix 


& ir | (1.40) 


to RSF. If the RSF is ordered so that its stable eigenvalues (there are exactly n of them under certain 
standard assumptions) are in the upper left corner, the corresponding first n vectors of the orthogonal 
matrix, which effects the reduction, forms a basis for the stable eigenspace from which the nonnegative 
definite solution X is then easily found. 

Extensions to the basic Schur method have been made [14, p. 562, 573], which were prompted by the 
following situations: 


¢ Gin Equation 1.37 is of the form BR~'B", where R may be nearly singular, or G2 in Equation 1.38 
may be exactly or nearly singular. 

« A in Equation 1.38 is singular (A~! is required in the classical approach involving a symplectic 
matrix that plays a role analogous to Equation 1.40). 


This resulted in the generalized eigenvalue approach requiring the computation of a basis for the 
deflating subspace corresponding to the stable generalized eigenvalues. For the solution of Equation 1.37, 
the generalized eigenvalue problem is given by 


I 0 0 A 0 6B 
JO I O;-—|-Q -AT 0]; (1.41) 
00 0 0 B' R 
for Equation 1.38, the corresponding problem is 
I 0 0 A 0 —-G, 
r~] 0 AT O}-]| -Q I Oo |. (1.42) 
0 Gl 0 00 G& 


The extensions above can be generalized even further, as the following problem illustrates. Consider 
the optimal control problem 


1p te 
min A [ [x! Qx + 2x7 Su+ ul Ru] dt (1.43) 
0 


subject to 
Ex = Ax + Bu. (1.44) 


The Riccati equation associated with Equations 1.43 and 1.44 then takes the form 
E'XBR-'B' XE—(A—BR™!S')'XE—E'X(A—BR“'S')— Q+SR'S' =0 (1.45) 


or 


(E'XB+S)R71(B'XE+S")—A’XE—E'XA—Q=0. (1.46) 


This so-called “generalized” Riccati equation can be solved by considering the associated matrix pencil 


A 0 B E 0 0 
Q AT -~S}|—2x]0 EF oO}. (1.47) 
st Bt R 0 0 0 


Note that S in Equation 1.43 and E in Equation 1.44 are handled directly and no inverses appear. The 
presence of a nonsingular E in state-space models of the form Equation 1.44 adds no particular difficulty 
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to the solution process and is numerically the preferred form if E is, for example, near singular or even 
sparse. Similar remarks apply to the frequency response problem in Equations 1.29 and 1.30 and, indeed, 
throughout all of linear control and system theory. The stability and conditioning of these approaches are 
discussed in [14, p. 529, 573]. Other methods, including Newton’s method and iterative refinement, have 
been analyzed in, for example, [14, p. 517]. Numerical algorithms for handling Equations 1.41, 1.42, and 
1.47 and a large variety of related problems are described in [14, p. 421, 573]. A thorough survey of the 
Schur method, generalized eigenvalue/eigenvector extensions, and the underlying algebraic structure in 
terms of “Hamiltonian pencils” and “symplectic pencils” is included in [3,12]. 

Schur techniques can also be applied to Riccati differential and difference equations and to nonsym- 
metric Riccati equations that arise, for example, in invariant imbedding methods for solving linear two- 
point boundary value problems. 

As with the linear Lyapunov and Sylvester equations, satisfactory results have been obtained concerning 
condition of Riccati equations, a topic of great interest independent of the solution method used, be it a 
Schur-type method or one of numerous alternatives. We refer to [1] for a further discussion on this. 

A very interesting class of invariant-subspace-based algorithms for solving the Riccati equation and 
related problems uses the so-called matrix sign function. These methods, which are particularly attractive 
for very large-order problems, are described in detail in [14, p. 486] and the references therein. These 
algorithms are based on Newton’s method applied to a certain matrix equation. A new family of iterative 
algorithms of arbitrary order convergence has been developed in [14, p. 624]. This family of algorithms 
can be parallelized easily and yields a viable method of solution for very high-order Riccati equations. 


1.4.8 Pole Assignment and Observer Design 


Designing state or output feedback for a linear system, so that the resulting closed-loop system has a 
desired set of poles, can be considered an inverse eigenvalue problem. The state feedback pole assignment 
problem is as follows: Given a pair (A, B), one looks for a matrix F so that the eigenvalues of the matrix 


Ap=A+BF 


lie at specified locations or in specified regions. Many approaches have been developed for solving this 
problem. However, the emphasis is on numerically reliable methods and consideration of the numerical 
sensitivity of the problem, for example, see the papers in Part 6 of [14]. Special cases of the pole assignment 
problem arise in observer design [14, p. 407], and in deadbeat control for discrete-time systems (where 
A+ BF is required to be nilpotent) [14, p. 392]. The numerically reliable methods for pole assignment 
are based on reducing A to either an RSF, [14, p. 415], or to a Hessenberg or block Hessenberg (staircase) 
form [14, p. 357, 380]. The latter may be regarded a numerically robust alternative to the controllable 
or Luenberger canonical form whose computation is known to be numerically unreliable [14, p. 59]. 
For multi-input systems, the additional freedom available in the state-feedback matrix can be used for 
eigenvector assignment and sensitivity minimization for the closed-loop poles [14, p. 333]. There the 
resulting matrix Ar is not computed directly, but instead the matrices A and X of the decomposition 


Ap =XAX7! 


are computed via an iterative technique. The iteration aims to minimize the sensitivity of the placed 
eigenvalues dj; or to maximize the orthogonality of the eigenvectors ~;. 

Pole assignment by output feedback is more difficult, theoretically as well as computationally. Con- 
sequently, there are a few numerically reliable algorithms available [14, p. 371]. Other works on pole 
assignment have been concerned with generalized state-space or descriptor systems. 

The problem of observer design for a given state-space system {A, B, C} is finding matrices T, Ax, and 
K so that 

TAx —AT=KC (1.48) 
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whereby the spectrum of Ax is specified. Because this is an underdetermined (and nonlinear) problem in 
the unknown parameters of T, Ax, and K, one typically sets T = I and Equation 1.48 then becomes 


Ax =A+KC, 


which is a transposed pole placement problem. In this case, the above techniques of pole placement 
automatically apply here. In reduced order design, T is nonsquare and thus cannot be equated to the 
identity matrix. One can still solve Equation 1.48 via a recurrence relationship when assuming Ax in 
Schur form [14, p. 407]. 


1.4.9 Robust Control 


In the last decade, there has been significant growth in the theory and techniques of robust control; 
see, for example, [6] and the references therein. However, the area of robust control is still evolving 
and its numerical aspects have just begun to be addressed [13]. Consequently, it is premature to survey 
reliable numerical algorithms in the area. To suggest the flavor of the numerical and computational issues 
involved, in this section we consider a development in robust control that has attracted a great deal 
of attention, the so-called Hoo approach. Hoo and the related structured singular value approach have 
provided a powerful framework for synthesizing robust controllers for linear systems. The controllers are 
robust, because they achieve desired system performance despite a significant amount of uncertainty in 
the system. 

In this section, we denote by R(s)"*™ the set of proper real rational matrices of dimension n x m. The 
Hg norm of a stable matrix G(s) € R(s)"*™ is defined as 


G(s) loo = SUP Omax[G(jw)], (1.49) 
weR 
where Omax[-] denotes the largest singular value of a (complex) matrix. Several iterative methods are 
available for computing this norm. In one approach, a relationship is established between the singular 
values of G(jw) and the imaginary eigenvalues of a Hamiltonian matrix obtained from a state-space 
realization of G(s). This result is then used to develop an efficient bisection algorithm for computing the 
Ho norm of G(s). 
To describe the basic Hoo approach, consider a linear, time-invariant system described by the state- 
space equations 


x(t) = Ax(t) + By w(t) + Bou(t), 


z(t) — C, x(t) + Dy, w(t) + D,2u(t), (1.50) 
y(t) = Cox(t) + Dz w(t) + Dy u(t), 


where x(t) € R” denotes the state vector; w(t) € R™! is the vector of disturbance inputs; u(t) € R’ is the 
vector of control inputs; z(t) € R?! is the vector of error signals; and y(t) € R?? is the vector of measured 


: : : ; w Z|. 
variables. The transfer function relating the inputs * to the outputs H is 


| Gils) Gia(s) 

ee lat ee (1.51) 
_— {Du Diy Ci eee 
. EB a 2 a (sl — A) [Bi By] . (1.52) 


Implementing a feedback controller defined by 


u= K(s)y (1.53) 
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where K(s) € R(s)"?*?2, we obtain the closed-loop transfer matrix Tzy(s) € R(s)?!*™ from the distur- 
bance w to the regulated output z 


Tew = Gi + Gi2K(UI — GK) ‘Gar. (1.54) 


Next, define the set K of all internally stabilizing feedback controllers for the system in Equation 1.50, 
that is, 
K := {K(s) € R(s)"*?? : Tzy(s) is internally stable}. 


Now let K(s) € K, and define 
¥:= IlTew(S)lloo- (1.55) 


Then the Hoo control problem is to find a controller K(s) € K that minimizes y. The optimal value of y 
is defined as 


Yopt -= pus I Tzw(s) loo: (1.56) 


The original formulation of this problem was in an input-output setting, and the early methods for 
computing Yopt used either an iterative search involving spectral factorization and solving the resulting 
Nehari problem or computed the spectral norm of the associated Hankel plus Toeplitz operator. In a 
state-space formulation for computing Yopt, promising from the viewpoint of numerical computation, 
the problem is formulated in terms of two algebraic Riccati equations that depend on a gain parameter 
y. Then, under certain assumptions [13], it can be shown that for a controller K(s) €K to exist so 
that || Tzwlloo < y, three conditions have to be satisfied, namely, stabilizing solutions exist for the two 
Riccati equations, and the spectral radius of the product of the solutions is bounded by y’. If these 
conditions are satisfied for a particular value of y, the corresponding controller K(s) can be obtained 
from the solutions of the Riccati equations. The optimal gain, yop, is the infimum over all suboptimal 
values of y such that the three conditions are satisfied. 

The approach above immediately suggests a bisection-type algorithm for computing Yopt. However, 
such an algorithm can be very slow in the neighborhood of the optimal value. To obtain speedup near 
the solution, a gradient approach is proposed in [13]. The behavior of the Riccati solution as a function 
of y is used to derive an algorithm that couples a gradient method with bisection. It has been pointed 
out in [13] that the Riccati equation can become ill-conditioned as the optimal value of y is approached. 
It is therefore recommended in [13] that, instead of computing the Riccati solutions explicitly, invariant 
subspaces of the associated Hamiltonian matrices should be used. 


1.5 Mathematical Software 
1.5.1 General Remarks 


The previous sections have highlighted some topics from numerical linear algebra and their application 
to numerical problems arising in systems, control, and estimation theory. These problems represent only 
a very small subset of numerical problems of interest in these fields but, even for problems apparently 
“simple” from a mathematical viewpoint, the myriad of details that constitute a sophisticated implemen- 
tation become so overwhelming that the only effective means of communicating an algorithm is through 
mathematical software. Mathematical or numerical software is an implementation on a computer of an 
algorithm for solving a mathematical problem. Ideally, such software would be reliable, portable, and 
unaffected by the machine or system environment. 

The prototypical work on reliable, portable mathematical software for the standard eigenproblem 
began in 1968. EISPACK, Editions I and II [7,15], were an outgrowth of that work. Subsequent efforts of 
interest to control engineers include LINPACK [5] for linear equations and linear least-squares problems, 
FUNPACK (Argonne) for certain function evaluations, MINPACK (Argonne) for certain optimization 
problems, and various ODE and PDE codes. High-quality algorithms are published regularly in the 
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ACM Transactions on Mathematical Software. LAPACK, the successor to LINPACK and EISPACK, is 
designed to run efficiently on a wide range of machines, including vector processors, shared-memory 
multiprocessors, and high-performance workstations. 

Technology to aid in developing mathematical software in Fortran has been assembled as a package 
called TOOLPACK. Mechanized code development offers other advantages with respect to modifications, 
updates, versions, and maintenance. 

Inevitably, numerical algorithms are strengthened when their mathematical software is portable, 
because they can be used widely. Furthermore, such a software is markedly faster, by factors of 10 to 50, 
than earlier and less reliable codes. 

Many other features besides portability, reliability, and efficiency characterize “good” mathematical 
software, for example, 


¢ High standards of documentation and style so as to be easily understood and used 

« Ease of use; ability of the user to interact with the algorithm 

¢ Consistency/compatibility/modularity in the context of a larger package or more complex problem 

¢ Error control, exception handling 

¢ Robustness in unusual situations 

¢ Graceful performance degradation as problem domain boundaries are approached 

« Appropriate program size (a function of intended use, e.g., low accuracy, real-time applications) 

¢ Availability and maintenance 

« “Tricks” such as underflow-/overflow-proofing, if necessary, and the implementation of column- 
wise or rowwise linear algebra 


Clearly, the list can go on. 

What becomes apparent from these considerations is that evaluating mathematical software is a 
challenging task. The quality of software is largely a function of its operational specifications. It must 
also reflect the numerical aspects of the algorithm being implemented. The language used and the com- 
piler (e.g., optimizing or not) for that language have an enormous impact on quality, perceived and real, 
as does the underlying hardware and arithmetic. Different implementations of the same algorithm can 
have markedly different properties and behavior. 

One of the most important and useful developments in mathematical software for most control engi- 
neers has been very high-level systems such as MATLAB® and Scilab. These systems spare the engineer 
the drudgery of working at a detailed level with languages such as Fortran and C, and they provide a large 
number of powerful computational “tools” (frequently through the availability of formal “toolboxes”). 
For many problems, the engineer must still have some knowledge of the algorithmic details embodied in 
such a system. 


1.5.2 Mathematical Software in Control 


Many aspects of systems, control, and estimation theory are at the stage from which one can start the 
research and design necessary to produce reliable, portable mathematical software. Certainly, many of the 
underlying linear algebra tools (e.g., in EISPACK, LINPACK, and LAPACK) are considered sufficiently 
reliable to be used as black, or at least gray, boxes by control engineers. An important development in this 
area is the SLICOT library, which is described in [1, p. 60]. Much of that theory and methodology can and 
has been carried over to control problems, but this applies only to a few basic control problems. Typical 
examples are Riccati equations, Lyapunov equations, and certain basic state-space transformations and 
operations. Much of the work in control, particularly design and synthesis, is simply not amenable to nice, 
“clean” algorithms. The ultimate software must be capable of enabling a dialogue between the computer 
and the control engineer, but with the latter probably still making the final engineering decisions. 
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1.6 Concluding Remarks 


Several numerical issues and techniques from numerical linear algebra together with a number of impor- 
tant applications of these ideas have been outlined. A key question in these and other problems in systems, 
control, and estimation theory is what can be computed reliably and used in the presence of parameter 
uncertainty or structural constraints (e.g., certain “hard zeros”) in the original model, and rounding errors 
in the calculations. However, because the ultimate goal is to solve real problems, reliable tools (mathemat- 
ical software) and experience must be available to effect real solutions or strategies. The interdisciplinary 
effort during the last few decades has significantly improved our understanding of the issues involved in 
reaching this goal and has resulted in some high-quality control software based on numerically reliable and 
well-tested algorithms. This provides clear evidence of the fruitful symbiosis between numerical analysis 
and numerical problems from control. We expect this symbiotic relationship to flourish, as control engi- 
neering realizes the full potential of the computing power becoming more widely available in multiprocess- 
ing systems and high-performance workstations. However, as in other applications areas, software contin- 
ues to act as a constraint and a vehicle for progress. Unfortunately, high-quality software is very expensive. 

In this chapter, we have focused only on dense numerical linear algebra problems in systems and 
control. Several related topics that have not been covered here are, for example, parallel algorithms, 
algorithms for sparse or structured matrices, optimization algorithms, ODE algorithms, algorithms for 
differential-algebraic systems, and approximation algorithms. These areas are well established in their 
own right, but for control applications a lot of groundwork remains undone. The main reason we have 
confined ourselves to dense numerical linear algebra problems in systems and control is that, in our 
opinion, this area has reached a mature level where definitive statements and recommendations can be 
made about various algorithms and other developments. 
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2.1 Introduction 


In this chapter we will introduce the basic building blocks necessary to understand linear time-invariant, 
multivariable systems. We will examine solutions of linear systems in both the time domain and frequency 
domain. An important issue is our ability to change the system’s response by applying different inputs. We 
will thus introduce the concept of controllability. Similarly, we will introduce the concept of observability 
to quantify how well we can determine what is happening internally in our model when we can observe 
only the outputs of the system. An important issue in controllability and observability is the role of zeros. 
We will define them for multivariable systems and show their role in these concepts. Throughout, we will 
introduce the linear algebra tools necessary for multivariable systems. 


2.2 Unforced Linear Systems 


2.2.1 Eigenvectors and Eigenvalues 


Given a matrix A € R”™", the eigenstructure of A is defined by n complex numbers );. When the j; are 
all different, each i; has corresponding vectors vj € C” and w; € C” so that 


Avi=hivs wHA=rywH; i=1...n (2.1) 


2-1 
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where w!! is the complex conjugate transpose of w. The complex numbers ; are called the eigenvalues 
of A. The vectors ; are called the right eigenvectors, and the vectors w; are called the left eigenvectors. 
Notice that any multiple of an eigenvector is also an eigenvector. 

The left and right eigenvectors are mutually orthogonal, that is, they satisfy the property 


1 ifi=j 


0 ififj C2) 


A 
why =a 2 


Weassume throughout that the eigenvalues are distinct; that is, they are all different. The case where eigen- 
values repeat is much more complicated, both theoretically and computationally. The case of repeated 
eigenvalues is covered in Kailath [1]. 

One other formula useful for describing linear systems is the dyadic formula. This formula shows how 
the matrix A can be formed from its eigenvalues and eigenvectors. It is given by 


n 
A= > riviwtt (2.3) 
i=l 
2.2.2 The Matrix Exponential 
The matrix exponential, e4, is defined by 
A La hag 
Pah A pA Ps (2.4) 
egg 
_ = ak 
= ae (2.5) 
k=0 


The matrix exponential solves the following matrix differential equation 
x(t) = Ax(t), x(0)=§ (2.6) 


The solution is 
x(t) = eA" (2.7) 


The matrix exponential can be calculated from the eigenstructure of the matrix A. If A has the 
eigenstructure as in Equation 2.1, then 


n n 
a= oS eNiyiwH and e4f= oe eit yiwHt (2.8) 
i=l i=1 


This can be seen by writing the matrix exponential using the infinite sum, substituting in the dyadic 
formula (Equation 2.3), and using Equation 2.2. 
Taking the Laplace transform of Equation 2.6, 
sx(s) — & = Ax(s) (2.9) 
where we have used the initial condition x(0) = &. Thus, the solution is 


x(s) = (sI — A) ls (2.10) 


Therefore, (sf — A)~! is the Laplace transform of eft, 
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2.2.3 Definition of Modes 


The solution to the unforced system Equation 2.6 can be written in terms of the eigenstructure of A as 


x(t) = ete =e vi(wi"8) (2.11) 


i=1 


The ith mode is e*"v;, defined by the direction of the right eigenvector v;, and the exponential associated 
with the eigenvalue ij. w!“€ is a scalar, specifying the degree to which the initial condition & excites the 
ith mode. 

Taking Laplace transforms, 


x0) => : -viwyte (2.12) 


From this equation, 


x(s) =(sf—A) 'g= 0 wht (2.13) 
i=1 4 


When the initial condition is equal to one of the right eigenvectors, only the mode associated with that 
eigenvector is excited. To see this, let § = v;. Then, using Equation 2.2, 


n 
x(t) = Doe yi(wily;) = edit y; (2.14) 
i=1 


A mode is called stable if the dynamics of the modal response tend to zero asymptotically. This is, 
therefore, equivalent to 


Re(Aj) < 0 (2.15) 


A system is said to be stable if all of its modes are stable. Thus, for any initial condition, x(t) — 0 if the 
system is stable. 


2.2.4 Multivariable Poles 
Consider the system 


x(t) = Ax(t) + Bu(t) (2.16) 
y(t) = Cx(t) + Du(t) (2.17) 


The multivariable poles are defined as the eigenvalues of A. Thus, the system is stable if all of its poles 
are strictly in the left half of the s-plane. The poles are therefore the roots of the equation 


det(sI — A) =0 (2.18) 
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2.3 Forced Linear Time-Invariant Systems 


2.3.1 Solution to Forced Systems 


We consider the dynamical system 


x(t) = Ax(t)+Bu(t), x(0)=€ (2.19) 
y(t) = Cx(t) + Du(t) (2.20) 
The solution x(t) to this equation is 
t 
x(t) = eA" +] eA Bu(t) dt (2.21) 
0 
This can be seen by differentiating the solution, and substituting in Equation 2.19. The output y(t) is thus 
t 
y(t) = Cx(t) + Du(t) = Cee + / CeA'—9 Bu(t) dt + Du(t) (2.22) 
0 
Using the eigenstructure of A, we can write 
y(t) = >> e# (Cvi)(wi"8) (2.23) 
i=l 
n t 
+ ys e™#(Cv;)(wH B) i eT y(t) dt + Du(t) (2.24) 
i=1 . 
Applying the Laplace Transform to the system Equations 2.19 and 2.20, 
y(s) = C(sI — A)~'Bu(s) + Du(s) (2.25) 
Using the eigenstructure of A, we can substitute for (sJ] — A)~ to get the Laplace Transform equation 
“\ CywhtB 
y(s) = py us) + Du(s) (2.26) 
. CviwHB , : : 
The matrix Guy called the residue matrix at the pole s = ij. 


Wecan see that w/’B is an indication of how much the ith mode is exited by the inputs, and Cy; indicates 
how much the ith mode is observed in the outputs. This is the basis for the concepts of controllability and 
observability, respectively. 


2.3.2 Controllability and Observability 


Controllability is concerned with how much an input affects the states of a system. The definition is 
general enough to handle nonlinear systems and time-varying systems. 


Definition 2.1: 


The nonlinear time-invariant (LTI) system 
x(t) =f[x(t), u(t)], x(0)=6, (2.27) 


is called completely controllable if, for any initial state & and any final state 0, we can find a piecewise, 
continuous bounded function u(t) for0 < t < T, T < 00, so that x(T) = 9. A system which is not completely 
controllable is called uncontrollable. 


Observability is defined similarly. 
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Definition 2.2: 


The nonlinear time-invariant system 


x(t) = f[x(t), u(t)], (0) =€ (2.28) 
y(t) = g[x(t), u()] (2.29) 


is observable if one can calculate the initial state & based upon measurements of the input u(t) and output 
y(t) for 0 <t < T, T < 00. A system which is not observable is called unobservable. 


For LTI systems, there are simple tests to determine if a system is controllable and observable. The ith 
mode is uncontrollable if, and only if, 


wi B=0 (2.30) 


Thus a mode is called uncontrollable if none of the inputs can excite the mode. The system is uncontrollable 
if it is uncontrollable from any mode. Thus, a system is controllable if 


wHB#O i=1,...,n (2.31) 
Similarly, the ith mode is unobservable if, and only if, 
Ci=0 (2.32) 


Thus a mode is called unobservable if we can not see its effects in any of the outputs. The system is 
unobservable if it is unobservable from any mode. Thus, a system is unobservable if 


Cy; #0, i=1,...,n (2.33) 


2.3.3 Other Tests for Controllability and Observability 


There is a simple algebraic test for controllability and observability. Let us define the controllability matrix 
M, as 
M,=[B AB A?B --- A™1B] (2.34) 


The system is controllable if, and only if, rank(M,) = n. Note that if the rank of M, is less than n, we do 
not know anything about the controllability of individual modes. We only know that at least one mode is 
uncontrollable. 

There is a similar test to determine the observability of the system. We form the observability matrix 
M, as 

C 
CA 


Meee CAE ly (2.35) 


can! 


The system is observable if, and only if, rank(M,) =n. Again, this test provides no insight into the 
observability of the modes of the system. It only determines whether or not the whole system is observable. 

We now have tests for controllability and observability. We will now relate the loss of controllability 
and observability to pole-zero cancellations. First, we need to define the concept of a zero for a multi-input 
multioutput system. 
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2.4 Multivariable Transmission Zeros 


There are several ways in which zeros can be defined for multivariable systems. The one we will examine 
is based on a generalized eigenvalue problem and has an interesting physical interpretation. Another 
approach to defining and calculating the zeros of MIMO systems can be found in Kailath [1]. 


2.4.1 Definition of MIMO Transmission Zeros 


To define the multi-input, multi-output (MIMO) transmission zeros, we will first assume that we have a 
system with the same number of inputs and outputs. This is referred to as a square system. We will later 
extend the definition to nonsquare systems. For square systems, we can represent the system in the time 
domain as 


x(t) = Ax(t) + Bu(t) (2.36) 
y(t) = Cx(t) + Du(t) (2.37) 


where x(t) € R”, u(t) € R™, and y(t) € R™. We can also write the transfer function matrix as 
G(s) = C(s] — A)"'B+D (2.38) 


where G(s) € C”*". Given this system, we have the following definition: 


Definition 2.3: 


The plant has a zero at the (complex) value z, if vectors €% € C" and uz € C™ exist which are not both zero, 
so that the solution to the equations 


X(t) = Ax(t) + Buze*!, x(0) =; (2.39) 

y(t) = Cx(t) + Du(t) (2.40) 
has the property that 

y(t)=0 Wt>0 (2.41) 


This property of transmission zeros is sometimes called transmission blocking. When zeros repeat, this 
definition still holds but a more complicated transmission blocking property also holds [2]. 

As an example, let us show that this definition is consistent with the standard definition of zeros for 
single-input, single-output systems. 


Example 2.1: 
Let us consider the following plant: 
s+1 
= 2.42 
g(s) =D (2.42) 
Then a state-space representation of this is 
x1 (t) = x2(t) (2.43) 
X2(t) = x2(t) + u(t) (2.44) 


y(t) = x1 (t) + x2(t) (2.45) 
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Let us define u(t) = 2e~—, so that Uz = 2 and z, = —1. Let us also define the vector E, as 
1 
f= a (2.46) 
Then 
X2(t) = x2(t)+2e~",  x2(0) = —1 (2.47) 
t 
=> x>(t)= —eb + é e 2etdt=-e! (2.48) 
0 
X1(t) = x2(t), x1 (0) = 1 (2.49) 
>x(t)=l1+e'-1=e! (2.50) 
Thus, 
y(t) = x1 (t) + x2(t) =0 (2.51) 


So we have confirmed that z = —1 is a transmission zero of the system. 


From the definition, we can see how transmission zeros got their name. If an input is applied at the 
frequency of the transmission zero in the correct direction (u;), and the initial condition is in the correct 
direction &;, then nothing is transmitted through the system. 


2.4.2 Calculation of Transmission Zeros 


To calculate the transmission zero, we can rewrite the definition of the transmission zero in matrix form as 


ZyIl—-A —B||&& _ {0 
Pt IRE es 
This is in the form of a generalized eigenvalue problem, typically written as 
Gyj = zZiMv; (2.53) 
wig = ziwiM (2.54) 


where z; is a generalized eigenvalue, with right and left generalized eigenvectors v; and wj;, respectively. 
The generalized eigenvalues are the roots of the equation 


det(zM — G) =0 (2.55) 


Note that if M is invertible, there are n generalized eigenvalues. Otherwise, there are less than n. 
Equation 2.52 is a generalized eigenvalue problem with 


A B I 0 
pe | u=(j 4 (2.56) 


Let us look at the implication of the generalized eigenvalue problem. Let z, be a transmission zero. 
Then it must be true that 


_, [al-A —B 
0 = det | Me | (2.57) 
If there are no common poles and zeros, then z;,J — A is invertible, and we can write 
0 = det(z,I — A) det[C(z,I — A) 'B+D] (2.58) 


= det(z,I — A) det[G(z)] (2.59) 
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Since we assume there are no common poles and zeros in the system, then det(z,J — A) £ 0, and so it 
must be true that 
det(G(z,)) = 0 (2.60) 


Thus, in the case that there are no common poles and zeros, the MIMO transmission zeros are the roots 
of Equation 2.60. To check for transmission zeros at the same frequencies as the poles, we must use the 
generalized eigenvalue problem. 

Let us now give a multivariable example. 


Example 2.2: 


Consider the system given by Equations 2.36 and 2.37, with the matrices given by 


—2 O 0 -1 0 
A=|0 -3 0 B=]-1 0 (2.61) 
0 0 -4 | 
1 0 0 1 0 
ies o 


Solving the generalized eigenvalue problem Equation 2 52, we find that there are two zeros, given by 


A= —1 Z22= —3 (2.63) 
2 0 
fi, =]|1 52=]| 1 (2.64) 


uy = i u2 = ay (2.65) 


Let us also check the transfer function calculation. The transfer function matrix for this system is 


given by 
1 
E 0 ] 
s+2 
G(s) = (2.66) 
12 1 
s+3 s+4 


To find transmission zeros at locations other than the poles of the system, we take the determinant 
and set it equal to zero to find 
0 = det(G(z)) = —*+! (2.67) 
(s+ 2)(s+ 4) 

Thus we find a transmission zero at z = —1. Notice that we have correctly found the transmission 
zero which is not at the pole frequency, but in this case we did not find the transmission zero at the 
same frequency as the pole at s = —3. It is also important to realize that, although one of the SISO 
transfer functions has a zero at s = —2, this is not a transmission zero of the MIMO system. 

Also notice that this frequency domain method does not give us the directions of the zeros. We 
need to use the generalized eigenvalue problem to determine the directions. 


2.4.3 Transmission Zeros for Nonsquare Systems 


When we have a nonsquare system, we differentiate between right zeros and left zeros. In the following, 
we will assume we have a system with n states, m inputs, and p outputs. 
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Definition 2.4: 


zt is a right zero of the system, 


x(t) = Ax(t) + Bu(t) (2.68) 
y(t) = Cx(t) + Du(t) (2.69) 


if vectors &% € C" and u, € C™ exist, both not zero, so that 


Z41—A —B]|&x _ {0 
ot colli =E 70 


Definition 2.5: 


Zz is a left zero of the system, 


x(t) = Ax(t) + Bu(t) (2.71) 
y(t) = Cx(t) + Du(t) (2.72) 
if it is a right zero of the system, 
x(t) = A’ x(t) + Cl u(t) (2.73) 
y(t) = B’ x(t) + D7 u(t) (2.74) 


In other words, z, is a left zero of our system if vectors nk € C" and y, € C? exist, both not zero, so that 
zI—-A —B 
[ne ve] | va =[0 0] (2.75) 


For square systems, the left and right zeros coincide; any frequency which is a right zero is also a 
left zero. 


2.5 Multivariable Pole-Zero Cancellations 


Now that we have defined the zeros of a multivariable system, we are in a position to describe pole-zero 
cancellations and what they imply in terms of controllability and observability. 

First, we give a SISO example, which shows that a pole-zero cancellation implies loss of controllability 
or observability. 


Example 2.3: 


Consider the system given by the equations 
xq(t)} [OTP] x(t) 0 
Ba a ; 4 | x(t) | H u(t) (2.76) 


yit)=[1 —1] E Al (2.77) 
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The transfer function for this system is given by 

—s+1 s—1 
st—1 (s+ 1)(s—1) 
Thus, there is a pole-zero cancellation at s = 1. To check for loss of controllability and observability, 


we will perform an eigenvalue decomposition of the system. The eigenvalues and associated left 
and right eigenvalues are given by 


(2.78) 


g(s) = 


se n=[4] w,=[05 —0.5] (2.79) 


MgQ=1 W= H w2=[05 0.5] (2.80) 

It is easy to verify that the dyadic decomposition Equation 2.3 holds. We now can use the modal tests 
for controllability and observability. We can see that 

Cy =2 wiB=-05 (2.81) 


This tells us that the first mode is both controllable and observable, as expected. For the second 
mode, 
Cv7=0 w3B=05 (2.82) 


We see that the second mode is controllable, but unobservable. The conclusion is that this particular 
state-space realization has an unobservable mode, but is controllable. 


Now let us examine what happens when we lose observability in more detail. We will assume we have 
the system described by Equations 2.36 through 2.38. Let us assume a pole with frequency i, and direction 
vz, that is, A has an eigenvalue 4; with associated eigenvalue vz. Thus 


(gl — A)v, = 0 (2.83) 
Let us also assume a (right) zero at frequency z, with direction given by [g; uj | that is, 
(zI — A)E, — Buy =0 (2.84) 
CE, + Duy, = 0 (2.85) 
If it is true that 4, = z, and &% = Bry with B any scalar, then 
Bu, =0 (2.86) 


If we assume that the columns of B are linearly independent (which says we don’t have redundant 
controls), then it must be true that u, = 0. However, with uz = 0, 


Cv, =0 = kth mode is unobservable (2.87) 
On the other hand, let us assume that the kth mode is unobservable. Then we know that 
Cy, =0 (2.88) 


where v, is the eigenvector associated with the kth mode. We will show that there must be a pole-zero 
cancellation. Let us choose u; = 0. If Xx is the eigenvalue associated with the kth mode, then we know 


Cyl — A)vy =O = (A441 —A)v, — Bug =0 (2.89) 
Cy. + Dur = 0 (2.90) 


Vk 
4 (2.91) 


Thus, d, is also a zero with direction 
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We have now proven the following theorem: 


Theorem 2.1: 


The kth mode, with eigenvalue \, and eigenvector vx, is unobservable if, and only if, \x is. a right transmission 
zero with direction given by Equation 2.91. 


Following the exact same steps, we show that loss of controllability implies a pole-zero cancellation. 
Using the same steps as before, we can prove the following theorem. 


Theorem 2.2: 


Let the kth mode have eigenvalue hx and left eigenvector wz. Then the kth mode is uncontrollable if, and 
only if, dy. is a left transmission zero with direction given by 


[wi 0] (2.92) 


We conclude with an example. 


Example 2.4: 


Consider the system Equations 2.19 and 2.20, with matrices 


—25 -05 05 1 
A=] 05 -15 05 B= 1 (2.93) 
1 1 -2 —2 
0 0 1 0 
eof Ope) 09 
A has the following eigenstructure: 
imo) 
M=-1 y=/1 wi =|! i 4 (2.95) 
1 2 2 2 
; 1 1 1 1 
KS? pre. lho Wee] sae 2.96 
2 2 : 2 E 5 4 (2.96) 
[<4 1 1 1 
=-3 y=] 0} wia=]- = - 2.97 
3 1 e 1 E 5 4 (2.97) 


Let us check for controllability and observability. 


2-12 Control System Advanced Methods 


(y= H wi B=0 (2.98) 
0 T 
Cv2 = Ey w,B=—1 (2.99) 
| T 
and Cv3= 0 w3B=2. (2.100) 


The conclusion is, therefore, that all modes are observable and that the first mode is uncontrollable, 
but the second and third modes are controllable. It is easy to verify that z = —1 is a left zero of this 
system, with direction 


1 1 1 

TE oF 

=!]_— —~ — Q QI. (2.101) 
[ny] E 2 2 

Thus there is a pole-zero cancellation at z = —1, which makes the system uncontrollable. 
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3.1 Introduction 


In this chapter, various fundamental elements of the theory of linear time-varying systems are studied 
in both the continuous-time and discrete-time cases. The chapter is a revised and updated version of 
Chapter 25 that appeared in the first edition of The Control Handbook. The revision includes material on 
the existence of coordinate transformations that transform time-varying system matrices into diagonal 
forms, and the connection to the concept of “dynamic” eigenvalues and eigenvectors. Also included in 
the revision is the use of a simplified expression derived in [11] for the feedback gain in the design of a 
state feedback controller in the single-input case based on the control canonical form. Generalizations 
to the multi-input multi-output case can be carried out using the results in [12]. The theory begins in 
Section 3.2 after the following comments. 

The distinguishing characteristic of a time-varying system is that the values of the output response 
depend on when the input is applied to the system. Time variation is often a result of system parameters 
changing as a function of time, such as aerodynamic coefficients in high-speed aircraft, circuit parameters 
in electronic circuits, mechanical parameters in machinery, and diffusion coefficients in chemical 
processes. Time variation may also be a result of linearizing a nonlinear system about a family of operating 
points and/or about a time-varying operating point. 


3-1 
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The values of the time-varying parameters in a system are often not known a priori, that is, before the 
system is put into operation, but can be measured or estimated during system operation. Systems whose 
parameters are not known a priori are often referred to as parameter varying systems, and the control 
of such systems is referred to as gain scheduling. Parameter varying systems and gain scheduling are 
considered in another chapter of this handbook, and are not pursued here. In some applications, time 
variations in the coefficients of the system model are known a priori. For example, this may be the case if 
the system model is a linearization of a nonlinear system about a known time-varying nominal trajectory. 
In such cases, the theory developed in this chapter can be directly applied. 

In this chapter, the study of linear time-varying systems is carried out in terms of input/output 
equations and the state model. The focus is on the relationships between input/output and state models, 
the construction of canonical forms, the study of the system properties of stability, controllability, and 
observability, and the design of controllers. Both the continuous-time and the discrete-time cases are 
considered. In the last part of the chapter, an example is given on the application to state observers and 
state feedback control, and examples are given on checking for stability. Additional references on the 
theory of linear time-varying systems are given in Further Reading. 


3.2 Analysis of Continuous-Time Causal Linear 
Time-Varying Systems 


Consider the single-input single-output continuous-time system given the input/output relationship 


t 
y(t) = / h(t, t)u(t) dt (3.1) 


where t is the continuous-time variable, y(t) is the output response resulting from input u(t), and h(t, t) 
is a real-valued continuous function of t and t. It is assumed that there are conditions on h(t, t) and/or 
u(t), which insure that the integral in Equation 3.1 exists. The system given by Equation 3.1 is causal 
since the output y(t) at time ¢ depends only on the input u(t) for t < t. The system is also linear since 
integration is a linear operation. Linearity means that if y(t) is the output response resulting from input 
u,(t), and y2(t) is the output response resulting from input u(t), then for any real numbers a and J, the 
output response resulting from input au; (t) + bu2(t) is equal to ay; (t) + by2(t). 

Let 8(t) denote the unit impulse defined by 8(t) = 0, t #0 and ee 3() d\ = 1 for any real number 
€ > 0. For any real number t), the time-shifted impulse 8(t — t;) is the unit impulse located at time t = f). 
Then from Equation 3.1 and the sifting property of the impulse, the output response y(f) resulting from 
input u(t) = 8(t — f)) is given by 


y(t) = iN A(t, t)u(t) dt = [ A(t, t)8(t — ty) dt = h(t, ty) 


Hence, the function h(t, t) in Equation 3.1 is the impulse response function of the system, that is, h(t, t) is 
the output response resulting from the impulse 8(t — t) applied to the system at time t. 
The linear system given by Equation 3.1 is time invariant (or constant) if and only if 


A(t +y,t+y)=A(t,t), for all real numbers t, t, y (3.2) 


Time invariance means that if y(t) is the response to u(t), then for any real number t,, the time-shifted 
output y(t — t;) is the response to the time-shifted input u(t — f)). Setting y = —t in Equation 3.2 gives 


h(t —1,0)=A(t,t), for all real numbers t, t (3.3) 


Hence, the system defined by Equation 3.1 is time invariant if and only if the impulse response function 
h(t, t) is a function only of the difference t — t. In the time-invariant case, Equation 3.1 reduces to the 
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convolution relationship 


y(t) = A(t) * u(t) = [ h(t — t)u(t) dt (3.4) 


where h(t) = h(t, 0) is the impulse response (i.e., the output response resulting from the impulse 8(¢) 
applied to the system at time 0). 

The linear system defined by Equation 3.1 is finite-dimensional or lumped if the input u(t) and the 
output y(t) are related by the nth-order input/output differential equation 


n—1 m 
IM OD+ DS ailQyOO = Yo bu (3.5) 
i=0 i=0 


where yl (t) is the ith derivative of y(t), u(t) is the ith derivative of u(t), and a;(t) and b;(t) are real-valued 
functions of ft. In Equation 3.5, it is assumed that m < n. The linear system given by Equation 3.5 is time 
invariant if and only if all coefficients in Equation 3.5 are constants, that is, aj(t) = a; and b;(t) = 0; for 
all i, where a; and 0; are real constants. 


3.2.1 State Model Realizations 


A state model for the system given by Equation 3.5 can be constructed as follows. First, suppose that 
m = 0 so that Equation 3.5 becomes 


n—-1 
YMC) + DS ai(thy (f) = bo(tu(t). (3.6) 
i=0 
Then defining the state variables 
x(th=y"%), i=1,2,...,0, (3.7) 
the system defined by Equation 3.6 has the state model 


x(t) = A(t)x(t) + B(t)u(t) (3.8) 
y(t) = Cx(t) (3.9) 


where the coefficient matrices A(t), B(t), and C are given by 


0 1 0 0 0 
0 0 1 0 0 
0 0 O. - a 0 0 
A(t) = . ; (3.10) 
0 0 0 tee 0 1 
ag(t) —a,(t) —az(t) +++ —@y—2(t) —an—1(t) 
0 
0 
0 
B(t)= : (3.11) 
0 
bo(t) 


C=[1 0 0 --- 0 OJ (3.12) 
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and x(t) is the n-dimensional state vector given by 
x(t) 
x2(t) 
x(t) = : 
Xn-1 (t) 
Xn(t) 


When m > 1 in Equation 3.5, the definition in Equation 3.7 of the state variables will not yield a state 
model of the form given in Equations 3.8 and 3.9. If m <n, a state model can be generated by first 
rewriting Equation 3.5 in the form 


n—1 m 
D*y(t) + D> D' [ay] = >) D' [Biu(t)] (3.13) 
i=0 i=0 


where D is the derivative operator and a;(t) and f;(t) are real-valued functions of t. The form of the 
input/output differential equation given by Equation 3.13 exists if a;(t) and b;(t) are differentiable a 
suitable number of times. If a;(t) are constants so that a;(t) = a; for all t, then a;(t) are constants and 
a(t) = aj,i=0,1,...,n—1. If bj(t) are constants so that b;(t) = 0; for all t, then 6;(t) are constants and 
Bi(t) = b;,i=0,1,...,m. 

When a;(t) and 6;(t) are not constants, a;(t) is a linear combination of a;(t) and the derivatives of a;(t) 
for j =n—i—1,n—i—2,...1, and B,(¢) is a linear combination of b;(t) and the derivatives of bj(t) for 
j=m—i-—1,m—i-—2,...1. For example, when n = 2, ao(t) and a1(t) are given by 


o(t) = ao(t) — ay (t) (3.14) 
y(t) = a(t) (3.15) 
When n = 3, a(t) are given by 
o(t) = ao(t) — ay (t) + ao(t) (3.16) 
0 (t) = ay (t) — 2a2(t) (3.17) 
2 (t) = a(t) (3.18) 


In the general case (n arbitrary), there is a one-to-one and onto correspondence between the coefficients 
a;(t) in the left-hand side of Equation 3.5 and the coefficients «;(t) in the left-hand side of Equation 3.13. 
Similarly, there is a one-to-one and onto correspondence between the coefficients in the right-hand side 
of Equation 3.5 and the right-hand side of Equation 3.13. 

Now given the system defined by Equation 3.5 written in the form of Equation 3.13, define the state 
variables 


Xn(t) = y(t) 
Xn—1(t) = Dxy(t) + On—1(t)Xn(t) = Bn—1(t)u(t) 
xn—2(t) = Dxn—1(t) + n2(€)%n(€) — Bn—2(E)u(t) (3.19) 


x1 (t) = Dx2(t) + 01 (t)xn(t) — Bi (t)u(t) 


where B;(t) = 0 for i > m. Then with the state variables defined by Equations 3.19, the system given by 
Equation 3.5 has the state model 


x(t) = A(t)x(t) + B(t)u(t) (3.20) 
y(t) = Cx(t) (3.21) 
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where the coefficient matrices A(t), B(t), and C are given by 


0 0 0 --+ 0 ~ap(t) 
100 --- 0  ~=aj,(t) 
0 10 +--+ 0 ~ar(t) 
A(t)=|. . (3.22) 
0 0 0 0 28s At 
0 0 0 -+ 1 =—ay—1(t) 
Bo(t) 
Bi (t) 
B2(t) 
B(t)= (3.23) 
Bn—2(t) 
Bn—1(t) 
and 
C=[0 0 + 0 1 (3.24) 


The state model with A(t) and C specified by Equations 3.22 and 3.24 is said to be in observer canonical 
form, which is observable as discussed below. Note that for this particular state realization, the row vector 
C is constant (independent of f). 

In addition to the state model defined by Equations 3.22 through 3.24, there are other possible state 
models for the system given by Equation 3.5. For example, another state model can be constructed in the 
case when the left-hand side of Equation 3.5 can be expressed in a Floquet factorization form (see [1]), so 
that Equation 3.5 becomes 


(D — pi(t))(D— pr(t))...(D—palt))y(t) = > bi(tu (3.25) 
i=0 


where again D is the derivative operator and the p;(t) are real-valued or complex-valued functions of the 
time variable t. For example, consider the n = 2 case for which 


(D — pi(t))(D — palt)) y(t) = (D — pil) @) — pay] 


; (3.26) 
(D — pilt))(D — p2(t)) y(t) = y(t) — [pilt) + poy) + [pi Op2(t) — prt) y) 

With n = 2 and m = 1, the state variables may be defined by 
x1(t) = y(t) — pa(t)y(t) — bi (t)u(t) (3.27) 
x2(t) = y(t) (3.28) 

which results in the state model given by Equations 3.20 and 3.21 with 

_{pitt) 0 

A(t) = | i oA (3.29) 
_ [ Bolt) — bi (t) + pr(e)br (t) 

B(t) = | b(t) (3.30) 


and 
c=[0 1] (3.31) 


3-6 Control System Advanced Methods 


In the general case given by Equation 3.25, there is a state model in the form of Equations 3.20 and 3.21 
with A(t) and C given by 


pitt) 0 0 O-- 0 0 
1 p(t) 0 O -+ 0 0 
0 1 p3(t) 0 --- 0 0 
A(th=|_ ; (3.32) 
0 0 0 0 Pity. 0 
0 0 0 0 1 pplt) 
c=[0 00: J (3.33) 


A very useful feature of the state model with A(t) given by Equation 3.32 is the lower triangular form of 
A(t). In particular, as discussed below, stability conditions can be specified in terms of p;(t), which can be 
viewed as “time-varying poles” of the system. However, in the time-varying case the computation of p;(t) 
based on the Floquet factorization given in the left-hand side of Equation 3.25 requires that nonlinear 
Riccati-type differential equations must be solved (see, e.g., [2,13]). The focus of [2] is on the computation 
of poles and zeros and the application to stability and transmission blocking, with the emphasis on time- 
varying difference equations. In [4], the authors generate a Floquet factorization given by the left-hand 
side of Equation 3.25 by utilizing successive Riccati transformations in a state-space setting. 

A significant complicating factor in the theory of factoring polynomial operators with time-varying 
coefficients is that in general there are an infinite number of different “pole sets”{p1(t), p(t), ..-, P(t}. 
This raises the question as to whether one pole set may be “better” in some sense than another pole set. 
In the case of linear difference equations with time-varying coefficients, it is shown in [2] that poles can 
be computed using a nonlinear recursion and that uniqueness of pole sets can be achieved by specifying 
initial values for the poles. For recent work on the factorization of time-varying differential equations 
by using ground field extensions, see [6]. As discussed below, in [7] the definition and computation of 
time-varying poles is pursued by using Lyapunov coordinate transformations in a state-space setting. 


3.2.2 The State Model 


For an m-input p-output linear n-dimensional time-varying continuous-time system, the general form of 
the state model is 


x(t) = A(t)x(t) + B(t)u(t) (3.34) 
y(t) = C(t)x(t) + D(t)u(t) (3.35) 


where Equation 3.34 is the state equation and Equation 3.35 is the output equation. In Equations 3.34 
and 3.35, A(t) is the n x n system matrix, B(t) is the n x m input matrix, C(t) is the p x n output matrix, 
D(t) is the p x m direct feed matrix, u(t) is the m-dimensional input vector, x(t) is the n-dimensional 
state vector, and y(t) is the p-dimensional output vector. The term D(t)u(t) in Equation 3.35 is of little 
significance in the theory, and thus D(t)u(t) is usually omitted from Equation 3.35, which will be done 
here. 

To solve Equation 3.34, first consider the homogeneous equation 


x(t) = A(t)x(t), t> to (3.36) 


with the initial condition x(fo) at initial time fo. For any A(t) whose entries are continuous functions 
of f, it is well known (see, e.g., [8]) that for any initial condition x(to), there is a unique continuously 
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differentiable solution of Equation 3.36 given by 
x(t) = P(t, to)x(to), t> to (3.37) 


where (tf, to) isan x n matrix function of t and fo, called the state-transition matrix. The state-transition 
matrix has the following fundamental properties: 


®(t, t) =I =n x nidentity matrix, for allt (3.38) 
M(t, t) = P(t, t)) P(t), 1), for allt,,t,t (3.39) 
© '(t,1) = @(t,t), forallt,t (3.40) 
< w(t, t)=A(t)®(t,t), for allt,t (3.41) 
2 ote t)=—(t, t)A(t), forallt,t (3.42) 

T 

t 
det ®(t, t) = exp | tr[A(o)] ao] (3.43) 
t 


In Equation 3.43, “det” denotes the determinant and “tr” denotes the trace. Equation 3.39 is called the 
composition property. It follows from this property that ®(f, t) can be written in the factored form 


P(t, t) = P(t, 0) (0, t), for allt,t (3.44) 


Another important property of the state-transition matrix P(t, t) is that it is a continuously differentiable 
matrix function of ¢ and t. 
It follows from Equation 3.42 that the adjoint equation 


y(t) = —AT(t)y(t) (3.45) 


has state-transition matrix equal to (t,t), where again @(t,t) is the state-transition matrix for 
Equation 3.36 and superscript “T” denotes the transpose operation. 

If the system matrix A(t) is constant over the interval [t), t2], that is, A(t) = A, for all t € [t), J, then 
the state-transition matrix is equal to the matrix exponential over [t), f2]: 


(t,t) = e449 for allt, te [ty, t] (3.46) 


If A(t) is time varying and A(t) commutes with its integral over the interval [t,, f2], that is, 


t t 
A(t) | f A(o) ao] = | f A(o) ao| A(t), for allt, t € [t), f2] (3.47) 


then ®(f, t) is given by 


t 
@(t, t) = exp Il A(t) ax| , forallt,t€ [t,t] (3.48) 


Note that the commutativity condition in Equation 3.47 is always satisfied in the time-invariant case. 
It is also always satisfied in the one-dimensional (n = 1) time-varying case since scalars commute. Thus, 
(t,t) is given by the exponential form in Equation 3.48 when n= 1. Unfortunately, the exponential 
form for ®(t, t) does not hold for an arbitrary time-varying matrix A(t) when n > 1, and, in general, there 
is no known closed-form expression for P(t, t) when n > 1. However, approximations to ®(f, t) can be 
readily computed from A(t) by numerical techniques, such as the method of successive approximations 
(see, e.g., [8]). Approximations to P(t, t) can also be determined by discretizing the time variable t as 
shown next. 


3-8 Control System Advanced Methods 


Let k be an integer-valued variable, let T be a positive number, and let a;;(t) denote the i, j entry of the 
matrix A(t). Suppose that by choosing T to be sufficiently small, the absolute values |a;j(t) — aj(kT)| can 
be made as small as desired over the interval t € [kT, kT + T] and for all integer values of k. Then for a 
suitably small T, A(t) is approximately equal to A(KT) for t € [kT, kT + T], and from Equation 3.46 the 
state-transition matrix P(t, kT) is given approximately by 


O(t, kT) = ARDCKY for allt [kT, KT+T] and allk (3.49) 
Setting t = kT + T in Equation 3.49 yields 
O(kKT +7, kT) = ACD? for allk (3.50) 


The state-transition matrix ®(kT + T, kT) given by Equation 3.50 can be computed by using a param- 
eterized Laplace transform. To show this, first define the n x n matrix 


qk, t) = eAkDt (3.51) 
Then q(k, f) is equal to the inverse Laplace transform of 
Q(k,s) = [sf — A(kT)]7! (3.52) 


Note that the transform Q(k, s) is parameterized by the integer-valued variable k. A closed-form expression 
for q(k,t) as a function of t can be obtained by expanding the entries of Q(k,s) into partial fraction 
expansions and then using standard Laplace transform pairs to determine the inverse transform of 
the terms in the partial faction expansions. Then from Equation 3.51, setting t = T in q(k,t) yields 
qk, T) = ®(kT + T,kT). 

Again consider the general case with the state equation given by Equation 3.34. Given the state- 
transition matrix P(t, t), for any given initial state x(t9) and input u(t) applied for t > to, the complete 
solution to Equation 3.34 is 


x(t) = D(t, to)x(to) +f O(t, t)B(t)u(t) dt, t> to (3.53) 
to 
Then, when y(t) = C(t)x(t), the output response y(t) is given by 
y(t) = C(t) ®(E, to)x(to) + i C(t) ®(t, WB(t)u(t) dt, t> to (3.54) 
0 
If the initial time fo is taken to be —oo and the initial state is zero, Equation 3.54 becomes 
y(t) = th C(t) ®(t, t)B(t) u(t) dt (3.55) 
—00 
Comparing Equation 3.55 with the m-input, p-output version of Equation 3.1 reveals that 
Hew one UE) ‘ ; (3.56) 


where H(t, t) is the p x m impulse response function matrix. Inserting Equation 3.44 into Equation 3.56 
reveals that H(t, t) can be expressed in the factored form, 


A(t,t) = Hy(t)A2(t), t= (3.57) 
where 
H(t) = C(t)®(t,0) and H(t) = (0, t)B(t) (3.58) 


It turns out [8] that a linear time-varying system with impulse response matrix H(t, t) has a state realiza- 
tion given by Equations 3.34 and 3.35 with D(t) = 0 if and only if H(t, t) can be expressed in the factored 
form given in Equation 3.57. 
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3.2.3 Change of State Variables 


Suppose that the system under study has the n-dimensional state model 


x(t) = A(t)x(t) + B(t)u(t) (3.59) 
y(t) = C(t)x(t) (3.60) 


In the following development, the system given by Equations 3.59 and 3.60 will be denoted by the triple 
[A(t), B(t), C(t)]. 

Given any n x n invertible continuously differentiable matrix P(t), which is real or complex valued, 
another state model can be generated by defining the new state vector z(t) = P7!(t)x(t), where P~1(t) 
is the inverse of P(t). The matrix P~!(t) (or the matrix P(t)) is called the transformation matrix and the 
process of going from x(t) to z(t) is referred to as a coordinate transformation. The state variables for 
the new state model (i.e., the z;(t)) are linear combinations with time-varying coefficients of the state 
variables of the given state model. To determine the state equations for the new state model, first note 
that P(t)z(t) = x(t). Then taking the derivative of both sides of P(t)z(t) = x(t) gives 


P(t)z(t) + P(t)z(t) = x(t) (3.61) 
Inserting the expression for x(t) given by Equation 3.59 yields 
P(t)z(t) + P(t)z(t) = A(t)x(t) + B(t)u(t) (3.62) 
and replacing x(t) by P(t)z(t) in Equation 3.62 and rearranging terms results in 
P(t)z(t) = [A(t)P(t) — P(t) z(t) + B(t)u(t) (3.63) 
Finally, multiplying both sides of Equation 3.63 on the left by P~!(t) gives 
2(t) = [P| (f)A()P(t) — PH) P(t) + P(t) B(t)u(t) (3.64) 
and replacing x(t) by P(t)z(t) in Equation 3.60 yields 
y(t) = C(t)P(t)z(t) (3.65) 


The state equations for the new state model are given by Equations 3.64 and 3.65. This new model will 
be denoted by the triple [A(t), B(t), C(t)] where 


A(t) = P7!(t)A(t)P(t) — P(t) P(t) (3.66) 
B(t) =P! (t)B(t) (3.67) 
C(t) = C(t)P(t) (3.68) 


Multiplying both sides of Equation 3.66 on the left by P(t) and rearranging the terms yield the result that 
the transformation matrix P(t) satisfies the differential equation 


P(t) = A(#)P(t) — P(t)A(t) (3.69) 


The state models [A(t), B(t), C(t)] and [A(t), B(t), C(t)] with A(t), B(t), C(t) given by Equations 3.66 
through 3.68 are said to be algebraically equivalent. The state-transition matrix @(t, t) for [A(t), B(t), C(O] 
is given by 

@(t, t) = P-!(t) @(t, t) P(t) (3.70) 


where ®(t, t) is the state-transition matrix for [A(t), B(t), C(t)]. 
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Given an n-dimensional state model [A(t), B(t), C(t)] and any n x n continuous matrix function I(t), 
there is a transformation matrix P(t) that transforms A(t) into I(t), that is, A(f) = I(t). To show this, 
define P(t) by 

P(t) = ®(t, 0) ®(0, t) (3.71) 


where ®(t, t) is the state-transition matrix for 2(t) = I'(t)z(t). Note that P(t) is continuously differen- 
tiable since ®(t,0) and ®(0, t) are continuously differentiable, and P(t) is invertible since state-transition 
matrices are invertible. Then taking the derivative of both sides of Equation 3.71 gives 


P(t) = O(t, 0) B(0, t) + b(t, 0) B(0, t) (3.72) 
Using Equations 3.41 and 3.42 in Equation 3.72 yields 
P(t) = —®(t, 0) ®(0, NF (t) + A(t) @(t, 0) B(0, t) (3.73) 


Finally, inserting Equation 3.71 into Equation 3.73 results in Equation 3.69 with A(t) = I(t), and thus, 
P(t) transforms A(t) into I(t). 

This result shows that via a change of state, A(t) can be transformed to any desired continuous matrix 
I(t). The fact that any continuous system matrix A(t) can be transformed to any other continuous matrix 
raises some interesting issues. For example, suppose that the transformation z(t) = P—!(£)x(t) is defined 
so that the new system matrix A(t) is equal to a diagonal matrix A (t) with real or complex-valued functions 
d1i(t), X2(f),..-5 Xn(t) on the diagonal. Then from Equation 3.66, 


A(t) = P71(t)A(t) P(t) — P71 (t) P(t) (3.74) 
and multiplying both sides of Equation 3.74 on the left by P(t) gives 
P(t) A(t) = A(t)P(t) — P(t) (3.75) 


Now for i= 1,2,...,n, let y;(t) denote the ith column of the transformation matrix P(t). Then from 
Equation 3.75, it follows that 


Myi(t) = AMyilt) — vit), i= 1,2,...,n (3.76) 
Rearranging terms in Equation 3.76 gives 
[A(t) — (AW yi(t) = vit), i=1,2,...,n (3.77) 


where I is the n x n identity matrix. 

Equation 3.77 has appeared in the literature on linear time-varying systems, and has sometimes been 
used as the justification for referring to the );(t) as “dynamic” eigenvalues or poles, and the corresponding 
yi(t) as eigenvectors, of the time-varying system having system matrix A(t). But since Equation 3.77 holds 
for any continuous scalar-valued functions \j(t), \2(f),..-, An(t), or for any constants 41, A2,.--, An, the 
existence of d;(t) and y;(t) satisfying Equation 3.77 is not useful unless further conditions are placed on 
the transformation matrix P(t). For example, one can consider stronger notions of equivalence such as 
topological equivalence. This means that, in addition to being algebraically equivalent, the transformation 
matrix P(t) has the properties 


det P(t)| >co forall t (3.78) 
|p; (t)| <c, forallt and ij=1,2,...,n (3.79) 


where pj(t) is the i, j entry of P(t), and co and c) are finite positive constants. The conditions given 
in Equations 3.78 and 3.79 are equivalent to requiring that P(t) and its inverse P~!(t) be bounded 
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matrix functions of t. A transformation z(t) = P~!(t)x(t) with P(t) satisfying Equations 3.78 and 3.79 
is called a Lyapunov transformation. As noted in Section 3.2.4, stability is preserved under a Lyapunov 
transformation. 

If the transformation z(t) = P~'(t)x(t) is a Lyapunov transformation and puts the system matrix 
A(t) into the diagonal form A(t) = diag(X1(f), h2(t),..-, n(t)), then the ,(f) are of interest since they 
determine the stability of the system. However, in the general linear time-varying case, the existence of 
a Lyapunov transformation that puts A(f) into a diagonal form is a strong condition. A much weaker 
condition is the existence of a Lyapunov transformation that puts A(t) into an upper or lower triangular 
form (see, e.g., [7]). This is briefly discussed in Section 3.2.4. 


3.2.4 Stability 


Given a system with n-dimensional state model [A(t), B(t), C(t)], again consider the homogeneous 
equation 


x(t) = A(t)x(t), t> to (3.80) 


with solution 
x(t) = @(t, to)x(to), £> to (3.81) 
The system is said to be asymptotically stable if for some initial time fo, the solution x(t) satisfies the 


condition ||x(t)|| > 0 as t > oo for any initial state x(fo) at initial time fo. Here ||x(t)|| denotes the 
Euclidean norm of the state x(t) given by 


Ix( = VeT Ox) = 20) 4230) 4+ $200) (3.82) 
where x(t) = [x1 (t) x2(f)...xn(t)]?. A system is asymptotically stable if and only if 
| P(E, to)|| > 0 ast— oo (3.83) 
where || ®(f, fo) || is the matrix norm equal to the square root of the largest eigenvalue of ©! (t, ty) P(t, to), 
where t is viewed as a parameter. 
A stronger notion of stability is exponential stability, which requires that for some initial time fo, there 
exist finite positive constants c and > such that for any x(to), the solution x(t) to Equation 3.80 satisfies 
xe) < ce) |[x(to)I], > to (3.84) 


If the condition in Equation 3.84 holds for all to with the constants c and fixed, the system is said to 
be uniformly exponential stable. Uniform exponential stability is equivalent to requiring that there exists 
finite positive constants y and \ such that 


|®(t, )|| < ye *¢-Y for all t,t such that t > t (3.85) 


Uniform exponential stability is also equivalent to requiring that, given any positive constant 8, there 
exists a positive constant T such that for any fo and x(fo), the solution x(t) to Equation 3.80 satisfies 


|x(t)ll < 8 llx(to)|| fort > to +T (3.86) 


It follows from Equation 3.85 that uniform exponential stability is preserved under a Lyapunov trans- 
formation z(t) = P~!(t)x(t). To see this, let ®(t, t) denote the state-transition matrix for 2(t) = A(t)z(t), 
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where 
A(t) = P~!(t)A(t)P(t) — P~'(t)P(t) (3.87) 


Then using Equation 3.70 gives 
S(t, 0] <P Ol oe, DIP (3.88) 
But Equations 3.78 and 3.79 imply that 
|P(t)|| <M, and ||P -'(t)| <M (3.89) 
for some finite constants M; and Mp. Then inserting Equations 3.85 and 3.89 into Equation 3.88 yields 
| P(t, t)|| <yM\Mpe—*'-9 - for all t, tsuch that t > t (3.90) 


which verifies that 2(f) = A(t)z(t) is uniformly exponentially stable. 

If P(t) and P~!(t) are bounded only for t > t; for some finite t,, the coordinate transformation z(t) = 
P~!(t)x(t) preserves exponential stability of the given system. This can be proved using constructions 
similar to those given above. The details are omitted. 

When n = 1 so that A(t) = a(t) is a scalar-valued function of t, as seen from Equation 3.48, 


t 
P(t, to) = exp | / a(t) ar| : (3.91) 
to 
It follows that the system is asymptotically stable if and only if 
t 
/ a(o) do > —coast > co (3.92) 
to 
and the system is uniformly exponentially stable if a positive constant \ exists so that 
t 
/ a(o) do < —r(t— 1) for allt, tsuch that t > t (3.93) 
T 


When n > 1, if A(t) commutes with its integral for t > t; for some t, > to (see Equation 3.47), then 


t 
P(t, t)) = exp i/ A(o) ao] , t>t (3.94) 
ty 


and a sufficient condition for exponential stability (not uniform in general) is that the matrix function 


1 t 
= / A(o) do (3.95) 
t b 


be a bounded function of t and its pointwise eigenvalues have real parts < —f for t > t2 for some f2 > fy, 
where B is a positive constant. The pointwise eigenvalues of a time-varying matrix M(f) are the eigenvalues 
of the constant matrix M(t) for each fixed value of t viewed as a time-independent parameter. If A(t) 
is upper or lower triangular with p;(t), po(t),...,pPn(t) on the diagonal, then a sufficient condition for 
uniform exponential stability is that the off-diagonal entries of A(t) be bounded and the scalar systems 


Xi(t) = pi(t)x;(t) (3.96) 


be uniformly exponentially stable for i= 1,2,...,n. Note that the system matrix A(t) given by Equa- 
tion 3.32 is lower triangular, and thus, in this case, the system with this particular system matrix is 
uniformly exponentially stable if the poles p;(t) in the Floquet factorization given in the left-hand side of 
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Equation 3.25 are stable in the sense that the scalar systems in Equation 3.96 are uniformly exponentially 
stable. 

If there exists a Lyapunov transformation z(t) = P-1(t)x(t) that puts the system matrix A(t) into an 
upper or lower triangular form with pj(t), p2(t), ..., P(t) on the diagonal, then sufficient conditions for 
uniform exponential stability are that the off-diagonal entries of the triangular form be bounded and 
the scalar systems defined by Equation 3.96 be uniformly exponentially stable. The construction of a 
Lyapunov transformation that puts A(f) into an upper triangular form is given in [7]. In that paper, the 
authors define the set {pi(t), p(t), . .., Pu(t)} of elements on the diagonal of the upper triangular form to 
be a pole set of the linear time-varying system with system matrix A(f). 

Another condition for uniform exponential stability is that a symmetric positive-definite matrix Q(t) 
exists with c)I < Q(t) < c2I for some positive constants c1, c2, such that 


AT (1) Q(t) + Q(tA(t) + Q(t) < —c31 (3.97) 


for some positive constant cz. Here F < G means that F and G are symmetric matrices and G— F is 
positive semidefinite (all pointwise eigenvalues are > 0). This stability test is referred to as the Lyapunov 
criterion. For more details, see [8]. 

Finally, it is noted that if the entries of B(t) and C(t) are bounded, then uniform exponential stability 
implies that the system is bounded-input, bounded-output (BIBO) stable, that is, a bounded input u(t) 
always results in a bounded output response y(t). 


3.2.5 Controllability and Observability 


Given a system with the n-dimensional state model [A(t), B(t), C(t)], it is now assumed that the entries 
of A(t), B(t), and C(t) are at least continuous functions of t. The system is said to be controllable on the 
interval [fo,t,], where t; > to, if for any states xp and x,, a continuous input u(t) exists that drives the 
system to the state x(t;) = x; at time t = f) starting from the state x(to) = xo at time t = fo. 

Define the controllability Gramian which is the n x n matrix given by 


ty 
W(to, ti) = / ®(to, t)B(t)B! (t)®! (to, t) dt (3.98) 
to 


The controllability Gramian W (to, t1) is symmetric positive semidefinite and is the solution to the matrix 
differential equation 


d = Ty) T 
a ht) = AWE, H) + WE HA (t) — B(t)B’ (t), (3.99) 


W(t, ft) =0 


Then the system is controllable on [f, t;] if and only if W (fo, t) is invertible, in which case a continuous 
input u(t) that drives the system from x(to) = xo to x(t1) = x1 is 


u(t) = —B"(t)®" (to, t)W~ (to, f1) [xo -— P(t tii], to <t<t (3.100) 


There is a sufficient condition for controllability that does not require that the controllability Gramian 
be computed: Given a positive integer q, suppose that the entries of B(t) are q— 1 times continuously 
differentiable and the entries of A(t) are q— 2 times continuously differentiable, and define the n x m 
matrices 


Ko(t) = B(t) (3.101) 
Ki(t)=—-AMKi-1() + Ki-1(),_ i= 1,2...,q-1 (3.102) 
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Finally, let K(t) denotes the n x mq matrix whose ith block column is equal to Kj_;(¢), that is 
K(t) = [Ko(t) Ki(t)... Kg-1(0)] (3.103) 


Then a sufficient condition for the system [A(t), B(t), C(f)] to be controllable on the interval [to, 1] is 
that the n x mq matrix K(t) defined by Equation 3.103 has rank n for at least one value of t € [to, f1]. This 
condition was first derived in [9]. 

The rank condition on the matrix K(t) is preserved under a change of state variables. To show this, 
suppose the coordinate transformation z(t) = P~1(t)x(t) results in the state model [A(t), B(t), C(t)]. For 
this model, define 

K(t) = [Ko(t) Ki(t) ... Kg-1(t)] (3.104) 


where the K;(t) are given by Equations 3.101 and 3.102 with A(t) and B(t) replaced by A(t) and B(t), 
respectively. It is assumed that A(t) and B(t) satisfy the same differentiability requirements as given above 
for A(t) and B(f) so that K(t) is well defined. Then it follows that 


P-\(t)K(t) = K(t) (3.105) 


Now since P~!(t) is invertible for all t, P~!(t) has rank n for all t € [f, f,], and thus K(f) has rank n 
for some f, € [f, t1] if and only if K(t,) has rank n. Therefore, the system defined by [A(t), B(t), C(t)] is 
controllable on the interval [to, t] if and only if the transformed system [A(t), B(t), C(t)] is controllable 
on [fo, fy]. 

If the matrix K(t) has rank n for all t, the n x n matrix K(t)K! (f) is invertible for all t, and in this case 
Equation 3.105 can be solved for P~!(t). This gives 


P(t) = K(t)K" (t) [KK] (3.106) 


Note that Equation 3.106 can be used to compute the transformation matrix P~'(t) directly from the 
coefficient matrices of the state models [A(t), B(t), C(t)] and [A(t), B(t), C()]. 
Now suppose that the system input u(t) is zero, so that the state model is given by 


X(t) = A(t)x(t) (3.107) 
y(t) = C(t)x(t) (3.108) 


Inserting Equation 3.108 into the solution of Equation 3.107 results in the output response y(t) resulting 
from initial state x(to): 
y(t) = C(t) P(t, to)x(to), t > to (3.109) 


The system is said to be observable on the interval [fo, t)] if any initial state x(to) can be determined from 
the output response y(t) given by Equation 3.109 for ¢ € [fo, t)]. Define the observability Gramian which 
is the n x n matrix given by 


ty 
M(to.n)= f ©! (t, to)C! (t)C(t) P(t, to) dt (3.110) 
to 


The observability Gramian M (to, t,) is symmetric positive semidefinite and is the solution to the matrix 
differential equation 


d <2 48 = Sor 
Goro A’ (t)M(t, th) — M(t, t)A(t) — C* (C(t), (3.111) 


M(t, tH) =0 
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Then the system is observable on [to, t)] if and only if M(to, t1) is invertible, in which case the initial state 
x(to) is given by 


ty 
x(to) =M~'(to, nf ©! (t, t9)C! (t)y(t) dt (3.112) 
to 


Again given an m-input p-output n-dimensional system with state model [A(t), B(t), C(t)], the adjoint 
system is the p-input m-output n-dimensional system with state model [—A! (t), C!(t), B? (t)]. The adjoint 
system is given by the state equations 


y(t) = —A™(t)y(t) + CT (£)v(t) (3.113) 
n(t) = B" (t)y(t) (3.114) 


where y(t), v(t), and (t) are the state, input, and output, respectively, of the adjoint system. As noted 
above in Section 3.2.2, the state-transition matrix of the adjoint system [—A? (t), C? (t), B? (t)] is equal 
to 7 (t,t), where ®(t,t) is the state-transition matrix for the given system [A(t), B(t), C(t)]. Using 
this fact and the definition of the input and output coefficient matrices of the adjoint system given by 
Equations 3.113 and 3.114, it is clear that the controllability Gramiam of the adjoint system is identical 
to the observability Gramiam of the given system. Thus, the given system is observable on [fo, ti] if 
and only if the adjoint system is controllable on [to, t)]. In addition, the observability Gramiam of the 
adjoint system is identical to the controllability Gramiam of the given system. Hence, the given system is 
controllable on [fo, t)] if and only if the adjoint system is observable on [fo, f;]. 

A sufficient condition for observability is given next in terms of the system matrix A(t) and the output 
matrix C(t): Consider the system with state model [A(t), B(t), C(t)], and define the p x n matrices 


Lo(t) = C(t) (3.115) 
L(t) =L_,QAH4+L;10), i=1,2,...,q—-1 (3.116) 


where q is a positive integer. Consider the pq x n matrix L(t) whose ith block row is equal to Li_1(t), 
that is, 
Lo(t) 


Li(t) 
a ae (3.117) 


Tg-1(t) 
It is assumed that the entries of C(t) are q — 1 times differentiable and the entries of A(t) are q — 2 times 
differentiable, so that L(t) is well defined. Then as first shown in [9], the system is observable on [fo, t1] 
if the pq x n matrix L(t) defined by Equations 3.115 through 3.117 has rank n for at least one value of 
t € [to, ft]. 
Now consider the n x pq matrix U(t) = [Up(t) U(¢) ... Ug_1()] generated from the adjoint system 
[—A?(t), C! (t), B" (t)], where 


Up(t) = C7 (t) (3.118) 
Uj(t) = ATU; 1) + U1), i= 1,2,...,g-1 (3.119) 


Then from Equations 3.115 through 3.119, it is seen that the transpose U7 (t) of the n x pq matrix U(t) 
is equal to the pq x n matrix L(t) of the given system with state model [A(f), B(t), C(t)]. Since the transpose 
operation does not affect the rank of a matrix, the sufficient condition rank L(t) =n for observability 
of the system [A(t), B(t), C(t)] implies that the adjoint system [—A?(t), C’(t), B' (t)] is controllable. 
In addition, the rank condition for observability of the adjoint system implies that the given system is 
controllable. It also follows from the above constructions that the rank condition for observabiliy of the 
system [A(t), B(t), C(t)] is preserved under a coordinate transformation. 
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3.2.6 Control Canonical Form and Controller Design 


Now suppose that the system with state model [A(t), B(t), C(t)] has a single input (m= 1) so that the 
input matrix B(f) is an n-element column vector. Assuming that B(t) and A(t) can be differentiated an 
appropriate number of times, let R(t) denote the n x n matrix whose columns r;(f) are defined by 


r\(t) = Bt) (3.120) 
rii(t) = A(t)rj(t) —7;(t), i=1,2,...,n—-1 (3.121) 


Note that R(t) is a minor variation of the n x n matrix K(t) defined by Equations 3.101 through 3.103 
with q = n. In fact, since R(t) is equal to K(t) with a sign change in the columns, R(t) has rank n for all t 
if and only if K(t) has rank n for all t. Thus, R(t) is invertible for all t if and only if the matrix K(t) has 
rank n for all t. 

In some textbooks, such as [1], the matrix R(t) is called the controllability matrix of the system with 
state model [A(t), B(t), C(t)]. 

Assuming that R(f) is invertible for all t, define the n-element column vector 


n(t) = —R71(t)rng(t) (3.122) 


where ry+1(t) is defined by Equation 3.121 with i= n. The vector y(t) is invariant under any change of 
state z(t) = P~!(t)x(t). In other words, if Equations 3.120 through 3.122 are evaluated for the new state 
model [A(t), B(t), C(0)] resulting from the transformation z(t) = P!(£)x(t), Equation 3.122 will yield 
the same result for (t). In addition, if A(f) = A and B(t) = B where A and B are constant matrices, the 
vector 1 is constant and is given by 

n= [a9 a, ... An—i]" (3.123) 


where a; are the coefficients of the characteristic polynomial of A, that is, 


n—1 
det(sI — A) =s"+ DS as! (3.124) 
i=0 


Given the analogy with the time-invariant case, the vector y(t) given by Equation 3.122 can be viewed as 
a time-varying version of the characteristic vector of the system. 

If the n x n matrix R(t) with columns defined by Equations 3.120 and 3.121 is invertible for all f, 
there is a coordinate transformation z(t) = P~!(t)x(t), which converts [A(t), B(t), C(t)] into the control 
canonical form [A(t), B, C(t)] with A(t) and B given by 


0 1 0 0 0 
0 0 1 0 0 
0 0 0 es 0 0 
A() = ; ; (3.125) 
0 0 0 i 0 1 
Wo(t) Wilt) Walt) +++ —Wr-2(t) —Wr-i(t) 
0 
0 
0 
B=/. (3.126) 
0 
1 


This form was first derived in [10]. 
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A recursive procedure for computing the rows c;(t) of the matrix P~1(t) in the transformation z(t) = 
P~1(t)x(t) to the control canonical form was derived in [11], and is given by 


c1(t) = (en) R(t) (3.127) 
ci41(t) =¢(t)A(t)+¢(t), i=1,2,...,n-1 (3.128) 
where 
(é,)' = [0/0 O«- OF I] (3.129) 
As shown in [11], the entries ;(t) in the bottom row of A(t) are given by 
W(t) = Lholt) Wilt) --- Wa-1()] = —Cen (ACE) + en (t)) P(E) (3.130) 


Now consider the system in the control canonical form given by the state equation 
2(t) = A(t)z(t) + Bu(t) (3.131) 


where A(t) and B are defined by Equations 3.125 and 3.126. Then with the state feedback control u(t) = 
—g(t)z(t), where g(t) is an n-element row vector, from Equation 3.131 the state equation for the resulting 
closed-loop system is 

z(t) = [A(t) — Bg(t)]<(t) (3.132) 
Let d = [do d, ... dy—1] be an n-element row vector with any desired constants do, di, ..., dy_1. Then 
from the form of A(t) and B, it is clear that if g(t) is taken to be g(t) = d— W(t), then the resulting 
closed-loop system matrix A(t) — Bg(t) is equal to the matrix in the right-hand side of Equation 3.125 
with the elements —do, —d},..., —d,_) in the bottom row. Thus, the matrix A(f) — Bg(t) is constant and 
its characteristic polynomial is equal to s” + dys"! +--+ dys + do. The coefficients of the characteristic 
polynomial can be assigned to have any desired values do, di,...;dn—1. 

The state feedback u(t) = —g(t)z(t) in the control canonical form can be expressed in terms of the 
state x(t) of the given system [A(f), B(t), C(t)] by using the coordinate transformation z(t) = P-1(t)x(t). 
This results in 

u(t) = —g(t)z(t) = —g(t)P'(t)x(t) = —[d — (t)|P*(t) x(t) (3.133) 
Inserting the feedback u(t) given by Equation 3.133 into the state equation x(t) = A(t)x(t) + B(t)u(t) for 
the system [A(t), B(t), C(t)] yields the following equation for the resulting closed-loop system: 


x(t) = (A(t) — Bit) g(t) x(t) (3.134) 
where the feedback gain vector g(t) is given by 
g(t) = a()P'(1) =[d— p)P'() (3.135) 


By the above constructions, the coordinate transformation z(t) = P~1(t)x(t) transforms the system matrix 
A(t) — B(t)g(t) of the closed-loop system defined by Equation 3.134 into the system matrix A(t) — Bg(t) 
of the closed-loop system given by Equation 3.132. 

As shown in [11], the gain vector g(t) defined by Equation 3.135 can be computed without having to 
determine A(t) by using the following formula: 


n—1 
g(t)=— nat +>° de (3.136) 
i=0 

where the c;(f) are given by Equations 3.127 and 3.128 with i = 1,2, ...,n. Ifthe zeros of the characteristic 
polynomial of A(t) — Bg(t) are chosen so that they are located in the open left half of the complex plane 
and if the coordinate transformation z(t) = P~!(t)x(t) is a Lyapunov transformation, then the closed- 
loop system given by Equation 3.134 is uniformly exponentially stable. If P(t) and P~!(t) are bounded 
only for f > t; for some finite t;, then the closed-loop system is exponentially stable. In Section 3.4, an 
example is given which illustrates the computation of the feedback gain g(t). 
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3.3 Discrete-Time Linear Time-Varying Systems 


A discrete-time causal linear time-varying system with single-input u(k) and single-output y(k) can be 
modeled by the input/output relationship 


k 
y(k) = S> htk, j)u(j) (3.137) 


j=-00 


where k is an integer-valued variable (the discrete-time index) and h(k, j) is the output response resulting 
from the unit pulse 8(k — j) (where 8(k — j) = 1 fork =j and = 0 for k ¥ j) applied at time j. It is assumed 
that u(k) and/or h(k, j) is constrained so that the summation in Equation 3.137 is well defined. The system 
defined by Equation 3.137 is time invariant if and only if h(k, j) is a function of only the difference k — j, 
in which case Equation 3.137 reduces to the convolution relationship 


k 
y(k) = h(k) * u(k) = > h(k —j)u(j) (3.138) 
j=-oO 
where h(k — j) = h(k —j, 0). 
The system defined by Equation 3.138 is finite dimensional if the input u(k) and the output y(k) are 
related by the nth-order difference equation 


n—1 m 
yk +n) +> ailkyy(k +i) = S~ bi(kju(k +i) (3.139) 
i=0 i=0 


where m <n and the a;(k) and the b;(k) are real-valued functions of the discrete-time variable k. The 
system given by Equation 3.139 is time invariant if and only if all coefficients in Equation 3.139 are 
constants, that is, aj(k) = a; and b;(k) = 0; for all i, where a; and b; are constants. 

When m < n the system defined by Equation 3.139 has the n-dimensional state model 


x(k +1) = A(k)x(k) + B(k)u(k) (3.140) 
y(k) = Cx(k) (3.141) 
where 
000 --- 0 —ao(k) 
100: 0 —a\(k—1) 
0 10 - 0 —an(k — 2) 
A(k) =|. ; (3.142) 
0 0 0 -:-. O ge pS) 
0 0 0 +: Ll -ady_-\(k-—n+1) 
bo(k) 
bi(k — 1) 
b2(k — 2) 
B(k) = ‘ (3.143) 
bn-a(k—n+2) 
bn-1(k —n+1) 
and 
C=[0 0 --- 0 J 


where b;(k) = 0 for i > m. This particular state model is referred to as the observer canonical form. As in 
the continuous-time case, there are other possible state realizations of Equation 3.139, but these will not 
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be considered here. It is interesting to note that the entries of A(k) and B(k) in the observer canonical 
form are simply time shifts of the coefficients of the input/output difference Equation 3.139, whereas as 
shown above, in the continuous-time case this relationship is rather complicated. 


3.3.1 State Model in the General Case 


For an m-input p-output linear n-dimensional time-varying discrete-time system, the general form of the 
state model is 


x(k +1) = A(k)x(k) + B(k)u(k) (3.144) 
y(k) = C(k)x(k) + D(k)u(k) (3.145) 


where the system matrix A(k) is n x n, the input matrix B(k) is n x m, the output matrix C(k) is p x n, 
and the direct feed matrix D(k) is p x m. The state model given by Equations 3.144 and 3.145 may arise 
as a result of sampling a continuous-time system given by the state model 


x(t) = A(t)x(t) + B(t)u(t) (3.146) 
y(t) = C(t)x(t) + D(t)u(t) (3.147) 
If the sampling interval is equal to T, then setting t = kT in Equation 3.147 yields an output equation of the 


form in Equation 3.145, where C(k) = C(t)|;-¢7 and D(k) = D(t)|,;-¢7. To “discretize” Equation 3.146, 
first recall (see Equation 3.53) that the solution to Equation 3.146 is 


x(t) = P(t, to)x(to) +f o(t, t)B(t)u(t) dt, t> to (3.148) 
to 


Then setting t = kT + T and to = kT in Equation 3.148 yields 


kT+T 
x(kT + T) = O(KT + T, kT) x(kT) +f @(KT + T, t)B(t)u(t) dt (3.149) 
kT 
The second term on the right-hand side of Equation 3.149 can be approximated by 


kT+T 
| ®(kT + T, t)B(t) «| u(kT) 
k 


T 


and thus Equation 3.149 is in the form of Equation 3.144 with 


A(k) = ®(kT + T, kT) (3.150) 
kKT+T 
B(k) = / ®(kT + T, t)B(t) dt (3.151) 
kT 


Note that the matrix A(k) given by Equation 3.150 is always invertible since ®(kT + T,kT) is always 
invertible (see the property given by Equation 3.40). As discussed below, this implies that discretized or 
sampled data systems are “reversible.” 

From Equations 3.150 and 3.151 it is seen that the computation of A(k) and B(k) requires knowledge 
of the state-transition matrix ®(t, t) for t =kT+T and t € [kT,kT +T). As discussed in Section 3.3, 
if A(t) in Equation 3.146 is a continuous function of tf and the variation of A(t) over the intervals 
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(kT, kT + T] is sufficiently small for all k, then (kT + T, t) can be approximated by 
D(kT +7, 1) = AGMETHT 2) fort € [kT, kT +T) (3.152) 
and hence, A(k) and B(k) can be determined using 
A(k) = A@DT (3.153) 


kT+T 
B(k) = i eA(KT\KT+T—) Bq) dt (3.154) 
KT. 


Given the discrete-time system defined by Equations 3.144 and 3.145, the solution to Equation 3.144 is 


k-1 
x(k) = ®(k, ko)x(ko) + ys O(k,j+ IB ijulj), k>ko (3.155) 
j=ko 


where the n x n state-transition matrix ®(k, j) is given by 


not defined for k<ko 
@(k, ko) = ¢ I, k=ko (3.156) 
A(k — 1)A(k—2)...A(ko), k>ko 


It follows directly from Equation 3.156 that ®(k,ko) is invertible for k > kg only if A(k) is invert- 
ible for k > ko. Thus, in general, the initial state x(ko) cannot be determined from the relationship 
x(k) = ®(k, ko)x(ko). In other words, a discrete-time system is not necessarily reversible, although any 
continuous-time system given by Equations 3.146 and 3.147 is reversible since P(t, to) is always invertible. 
However, as noted above, any sampled data system is reversible. 

The state-transition matrix ®(k, kg) satisfies the composition property: 


®(k, ko) = O(k, ky) B(ky, ko), where ko < ky <k (3.157) 


and in addition, 
@O(k+1,ko) =A(kK)®(k, ko), k> ko (3.158) 


If A(k) is invertible for all k, ®(k, kg) can be written in the factored form 


@(k, ko) = Oy (k)P2(ko), k= ko (3.159) 
where 
A(k—1)A(k—2)....A(0), Red 
Oi (kK) = 27, k=0 (3.160) 


A(k—2)A7(k-— 3)... A7(-1), k <0 
A7!(0)A71(1)... A7 (kg — 1), ky > 0 
@2(ko) = 41, ky =0 (3.161) 
A(—1)A(—2)...A(ko), ky <0 
When the direct feed matrix D(k) in Equation 3.145 is zero, so that y(k) = C(k)x(k), the output response 
y(k) is given by 


k-1 
yk) = C(k)®(k, ko)x(ko) + D> C(k)(k,j + B({Ju(j), kK > ko (3.162) 
j=ko 
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If the initial time ko is set equal to —oo and the initial state is zero, Equation 3.162 becomes 
k-1 
y(k) = Ss C(k) ®(k,j + 1)B({)u(j) (3.163) 
j=-o 
Comparing Equation 3.163 with the m-input p-output version of the input/output Equation 3.137 reveals 
that 
C(k)®(k,j +1)BG), k>j 


3.164 
é ee (3.164) 


A(k,j) = | 
where H(k,j) is the p x m unit-pulse response function matrix. Note that if A(k) is invertible so that 
@(k, ko) has the factorization given in Equation 3.159, then H(k,j) can be expressed in the factored 
form as 

A(k,j) = [C(k) ®1 (kA) |[®2(7 + I)B(j)] fork > j. (3.165) 
As in the continuous-time case, this factorization is a fundamental property of unit-pulse response 
matrices H(k, j) that are realizable by a state model (with invertible A(k)). 


3.3.2. Stability 


Given an n-dimensional discrete-time system defined by Equations 3.144 and 3.145, consider the homo- 
geneous equation 

x(k+1)=A(k)x(k), k>ko. (3.166) 
The solution is 

x(k) = ®(k, ko)x(ko), k> ko (3.167) 


where ®(k, ko) is the state-transition matrix defined by Equation 3.156. 
The system is said to be asymptotically stable if for some initial time ko, the solution x(k) satisfies the 
condition ||x(k)|| > 0 as k — 0 for any initial state x(kg) at time ko. This is equivalent to requiring that 


| ®(k, ko) || > 0, ask — oo (3.168) 


The system is exponentially stable if for some initial time ko, there exist finite positive constants c and p 
with p < 1, such that for any x(kg) the solution x(k) satisfies 


Ilx(k)I| < cok—™ |Ix(ko)Il,  k > ko (3.169) 


If Equation 3.169 holds for all ko with the constants c and p fixed, the system is said to be uniformly 
exponentially stable. This is equivalent to requiring that there exist a finite positive constant y and a 
nonnegative constant p with p < 1 such that 


|| @(k,j)|| < ye", for all k, j such that k > j (3.170) 


Uniform exponential stability is also equivalent to requiring that given any positive constant 8, there 
exists a positive integer q such that for any ko and x(k), the solution to Equation 3.166 satisfies 


lx(k)|| < 8 |lx(ko)|l, k= ko+q (3.171) 
Uniform exponential stability is also equivalent to the existence of a finite positive constant B such that 
k 
Y> ®(k,i)| <P for allk,j such that k>j+1 (3.172) 
i=jHl 


Another necessary and sufficient condition for uniform exponential stability is that a symmetric positive- 
definite matrix Q(k) exists with cjI < Q(k) < cI for some positive constants c, and cz so that 


A? (kK)Q(k + 1)A(k) — Q(k) < —e31 (3.173) 


for some positive constant c3. 
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3.3.3 Controllability and Observability 


The discrete-time system defined by Equations 3.144 and 3.145 with D(k) = 0 will be denoted by the 
triple [A(k), B(k), C(k)]. The system is said to be controllable on the interval [ko, k,] with k, > ko if, for 
any states x9 and x), an input u(k) exists that drives the system to the state x(k,) = x, at time k= k, 
starting from the state x(ko) = xo at time k = ko. To determine a necessary and sufficient condition for 
controllability, first solve Equation 3.144 to find the state x(k,) at time k = k, resulting from state x(ko) 
at time k = ko and the input sequence u(ko), u(ko + 1),..., u(k,—1). The solution is 


x(k1) = ®(ky, ko)x(ko) + ®(ky, ko + 1)B(ko)u(ko) + ® (ky, ko + 2)B(ko + 1)u(ko + 1) 
faa ®(ky, ky 1)B(ky 2)u(ky 2) t B(ky 1)u(ky 1) (3.174) 


Let R(k1, ko) be the controllability (or reachability) matrix with n rows and (k; — ko)m columns defined by 
R(ko, ky) = [Blky — 1) ®(ky, ky — 1)Bky — 2)... (ky, ko + 2)B(ko + 1) ®(k1, ko + 1)B(ko)] (3.175) 

Then Equation 3.174 can be written the form 
x(k) = ®(ky, ko)x(ko) + R(ko ki) U (ko, ki) (3.176) 


where U(kg, ki) is the (kj — ko)m-element column vector of inputs given by 
T 
U(ko, ki) = [ua — 1) ul (ky —2) --- ul (ky +1) ul (ko)| (3.177) 


Now for any states x(k) = xo and x(k,) = x), from Equation 3.176, there is a sequence of inputs given by 
U(ko, ky) that drives the system from xg to x; if and only if the matrix R(ko, k,) has rank n. If this is the 
case, Equation 3.176 can be solved for U(ko, ky), giving 


U(ko, kx) = R" (ko, kx)| Rko, KR" (kos k1)] [x1 — ©(k1, ko) 0] (3.178) 


Hence, rank R(ko, k,) =n is a necessary and sufficient condition for controllability over the interval [ko, 
ky]. 

Given a fixed positive integer N, set ky = k—N-+1 and kj =k+1 in R(ko, k,), which results in the 
matrix R(k —N+1,k+1). Note that R(k —N+1,k+ 1) isa function of only the integer variable k and 
that the size of the matrix R(k — N + 1, k+ 1) is equal to n x Nm since kj — kp = N. The n x Nm matrix 
R(k —N-+1, k+1) will be denoted by R(k). By definition of the state-transition matrix ®(k, kg), R(k) 
can be written in the form 


R(k) = [Ro(k) Ri(k) ... Ry—-1(k)] (3.179) 

where the block columns R;(k) of R(k) are given by 
Ro(k) = B(k) (3.180) 
Rj(k) = A(K)R;_-1(kK—- 1), i=1,2,...,N—1 (3.181) 


The system is said to be uniformly N-step controllable if rank R(k) =n for all k. Uniformly N-step 
controllable means that the system is controllable on the interval [k — N+ 1, k +1] for all k. 
Now suppose that the system input u(k) is zero, so that the state model is given by 


x(k +1) = A(k)x(k) (3.182) 
yk) = C(k)x(k) (3.183) 


From Equations 3.182 and 3.183, the output response y(k) resulting from initial state x(ko) is given by 
y(k) = C(k) ®(k, ko)x(ko), k= ko (3.184) 


Then the system is said to be observable on the interval [ko,k,] if any initial state x(ko) = x9 can be 
determined from the output response y(k) given by Equation 3.184 for k = ko, kg + 1,...,k; — 1. Using 
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Equation 3.184 for k = ko to k = k,—1 yields 


yko) C(ko) x0 
y(ko + 1) C(ko + 1) ®(ko + 1, ko) x0 
= (3.185) 
y(ky — 2) C(ky — 2) (ky — 2, ko) xo 


y(ky — 1) C(ky — 1) ®(ky — 1, ko) xo 


The right-hand side of Equation 3.185 can be written in the form O(ko, k1)x9 where O(ko, k,) is the 
(k, — ko)p x n observability matrix defined by 


C(ko) 
C(ko + 1) ®(ko + 1, ko) 


O(ko, ky) = : (3.186) 
C(ky — 2) @(k, — 2, ko) 
C(ky — 1) @(k; — 1, ko) 
Equation 3.185 can be solved for any initial state xo if and only if rank O(ko, ki) =n, which is a nec- 


essary and sufficient condition for observability on [ko, k;]. If the rank condition holds, the solution of 
Equation 3.185 for x9 is 


-1 
x9 = [O" (ko, kr JOCK, kx) | OP (kos kx) ¥ (ko, kr) (3.187) 
where Y (ko, ky) is the (kj — ko)p-element column vector of outputs given by 


T 
¥ (kos kx) = [y" (ko) y" (ko +1) +-+y" (ka =2) 970 = )] (3.188) 


Given a positive integer N, setting kg =k and kj =k+N in O(ko, k,) yields the Np x n matrix 
O(k,k +N), which will be denoted by O(k). By definition of the state-transition matrix ®(k, ko), O(k) 
can be written in the form 


Oo(k) 
Oj (k) 
O(k) = d (3.189) 
Ov-1(8) 
where the block rows O;(k) of O(k) are given by 
Oo(k) = C(k) (3.190) 
Ok) = O;- (K+ 1)A(kK), i=1,2,...,.N—1 (3.191) 


The system is said to be uniformly N-step observable ifrank O(k) = n for all k. Uniformly N-step observable 
means that the system is observable on the interval [k, k + N] for all k. 


3.3.4 Change of State Variables and Canonical Forms 


Again, consider the discrete-time system with state model [A(k), B(k), C(k)]. For any n x n invertible 
matrix P(k), another state model can be generated by defining the new state vector z(k) = P-1(k)x(k). 
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The new state model is given by 


2(k + 1) = A(k)z(k) + B(k)u(k) (3.192) 
y(k) = C(k)z(k) (3.193) 
where 
A(k) = P'(k + 1)A(k)P(k) (3.194) 
B(k) = P-+(k + 1)B(k) (3.195) 
C(k) = C(k)P(k) (3.196) 


The state-transition matrix ®(k, ko) for the new state model is given by 
B(k, ko) = P~'(k) ®(k, ko) P(ko) (3.197) 


where ®(k, ko) is the state-transition matrix for [A(k), B(k), C(k)]. The new state model, which will be 
denoted by [A(k), B(k), C(k)], and the given state model [A(k), B(k), C(k)] are said to be algebraically 
equivalent. 

Given an n-dimensional state model [A(k), B(k), C(k)] and any n x n invertible matrix function I'(k), 
if A(k) is invertible, there is an invertible coordinate transformation matrix P(k) for k > ko, which trans- 
forms A(k) into '(k) for k > ko, that is, A(k) = I'(k), k > ko. To show this, define P(k) by the matrix 
difference equation 

P(kK+1L =A(kK)P(KAIT (kK), ke > ko (3.198) 


with initial condition P(ko) = I = n x n identity matrix. Then multiplying both sides of Equation 3.198 
on the right by P'(k) and multiplying the resulting equation on the left by P~!(k + 1) yields Equation 3.194 
with A(k) = I'(k) for k > ko. This result shows that an invertible matrix A(k) can be put into a diagonal 
form via a coordinate transformation with any desired nonzero functions or constants on the diagonal. 
Thus, as in the continuous-time case, there is no useful generalization of the notion of eigenvalues 
and eigenvectors in the time-varying case, unless additional conditions are placed on the coordinate 
transformation. For example, one can require that the transformation be a Lyapunov transformation, 
which means that both P(k) and its inverse P~!(k) are bounded matrix functions of the integer variable 
k. It follows from Equation 3.197 that uniform exponential stability is preserved under a Lyapunov 
transformation. 

Suppose that the systems [A(k), B(k), C(k)] and [A(k), B(k), C(k)] are algebraically equivalent and let 
R(k) denote the n x Nm controllability matrix for the system [A(k), B(k), C(k)], where R(k) is defined by 
Equations 3.179 through 3.181. Similarly, for the system given by [A(k), B(k), C(k)] define 


R(k) = [Ro(k) Ri(k) --. Rn—1(4)] (3.199) 


where the R;(k) are given by Equations 3.180 and 3.181 with A(k) and B(k) replaced by A(k) and B(k), 
respectively. Then the coordinate transformation P~!(k) is given by 


P—'(k+1)R(k) = R(k) (3.200) 


If the system [A(k), B(k), C(k)] is uniformly N-step controllable, R(k) has rank n for all k, and the n x n 
matrix R(k)R! (k) is invertible. Thus, Equation 3.200 can be solved for P7!(k+ 1), which gives 


PTU(k+1) =R(AR  (KIR(OR (KT! (3.201) 


It follows from Equation 3.200 that uniform N-step controllability is preserved under a change of state 
variables. 
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Now suppose that the n-dimensional system with state model [A(k), B(k), C(k)] is uniformly N-step 
controllable with N =n and that the system has a single input (m= 1) so that B(k) is an n-element 
column vector and R(k) isan x n invertible matrix. Define 


Rn(k) = A(k)Rn—1(k — 1) (3.202) 
n(k) = —R”'(k)Rn(k) (3.203) 


where R,—1(k) is the column vector defined by Equation 3.181 with i= n—1. The n-element column 
vector y(k) defined by Equation 3.203 is invariant under any change of state z(k) = P-1(k)x(k), and in 
the time-invariant case, y is constant and is given by 


n=[a a +++ aya)’ (3.204) 


where the a; are the coefficients of the characteristic polynomial of A. 
Given y(k) defined by Equation 3.203, write n(k) in the form 


n(k) =[mo(k) nik) +++ mn-1(k)] (3.205) 


Then as proved in [3], there is a transformation P(k) which converts [A(k), B(k), C(k)] into the control 
canonical form [A(k), B, C(k)], with A(k) and B given by 


0 1 0 0 0 
0 0 1 0 0 
- 0 0 0 Yat 0 0 
Akky=| (3.206) 
0 0 0 tee 0 1 
no(k) —ni(k+1) —n2(k+2) +--+» —Ny-2(k+n—2) —y-1(k+n-—1) 
0 
0 
B= (3.207) 
0 
1 


The transformation matrix P~!(k) that yields the control canonical form can be determined using Equa- 
tion 3.201. As in the continuous-time case, the control canonical form can be used to design a state 
feedback control which results in a uniformly exponentially stable closed-loop system if the coordinate 
transformation defined by P-l(k) isa Lyapunov transformation. If P(k) and P~!(k) are bounded only 
for k > k, for some finite k, the resulting closed-loop system is exponentially stable. The details are a 
straightforward modification of the continuous-time case, and thus are not pursued here. 


3.4 Applications and Examples 


In Section 3.4.1, an example is given on the construction of canonical forms and the design of observers 
and controllers. 
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3.4.1 Observer and Controller Design 


Consider the single-input single-output linear time-varying continuous-time system given by the 
input/output differential equation 


H(t) Fe H(t) + y(t) = u(t) (3.208) 


To determine the state model, which is in observer canonical form, write Equation 3.208 in the form 


H(t) + Dau (t)y(t)] + ao (t)y(t) = X(t) (3.209) 
In this case, 
ay(t)=e and a(t)=1+e? (3.210) 
Then from Equations 3.22 through 3.24, the observer canonical form of the state model is 
xi(t)] [0 -1—e*] [xi(t) 0 
E a = ‘ me Wee +]°] uo (3.211) 
x1 (t) 
t)=)]0 1 3.212 
yo=[o [20 (3.212) 
The state variables x, (t) and x2(t) in this state model are given by 
x(t) = p(t) +e“ y(t) — u(t) (3.213) 
x2(t) = y(t) (3.214) 


If the output y(t) of the system can be differentiated, then x, (t) and x(t) can be directly determined 
from the input u(t) and the output y(t) by using Equations 3.213 and 3.214. In practice, however, 
differentiation of signals should be avoided, and thus directly determining x; (t) using Equation 3.213 is 
usually not viable. As discussed next, by using a state observer, the state x(t) can be estimated without 
having to differentiate signals. Actually, in this particular example it is necessary to estimate only x; (¢) 
since the output y(t) = x(t) is known, and thus a reduced-order observer could be used, but this is not 
considered here. 

An observer for the state x(t) is given by 


d 
ae = A(t)x(t) + HL) — C()X(t)] + BO)u(t) (3.215) 


where H(t) is the n-element observer gain vector and X(t) is the estimate of x(t). With the estimation 
error e(t) defined by e(t) = x(t) — x(t), the error is given by the differential equation 


a(t) = [A(t) —H(t)C(t)] e(t), t >t (3.216) 


with initial error e(fo) at initial time to. The objective is to choose the gain vector H(t) so that, for any 
initial error e(t), |le(t)|| — 0 as t > oo, with some desired rate of convergence. 
For the system given by Equations 3.211 and 3.212, the error Equation 3.216 is 


oy [0 —l-et-hi@) 
e(t)= i et — y(t) e(t) (3.217) 
where H(t) =[hy(t) ho(t)|". From Equation 3.217, it is obvious that by setting 
h(t) =m) —1—e (3.218) 
(th =m—e* (3.219) 


where mp and m, are constants, the coefficient matrix on the right-hand side of Equation 3.217 is constant, 
and its eigenvalues can be assigned by choosing mo and my). Hence, any desired rate of convergence to 
zero can be achieved for the error e(t). 
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The estimate x(t) of x(t) can then be used to realize a feedback control law of the form 
u(t) = —g(t)x(t) (3.220) 


where g(f) is the feedback gain vector. The first step in pursuing this is to consider the extent to which the 
system can be controlled by state feedback of the form given in Equation 3.220 with x(t) replaced by x(t); 
in other words, the true system state x(t) is assumed to be available. In particular, we can ask whether or 
not there is a gain vector g(t) so that with u(t) = —g(t)x(t), the state of the resulting closed-loop system 
decays to zero exponentially with some desired rate of convergence. This can be answered by attempting 
to transform the state model given by Equations 3.211 and 3.212 to control canonical form. Following 
the procedure given in Section 3.2, the steps are as follows. 

Let R(t) denote the 2x2 matrix whose columns r;(t) are defined by Equations 3.120 and 3.121 with 
n = 2. This yields 


rj(t) = B(t) = H (3.221) 
; -l-e! 
r(t) = A(t)n(t) +n (t) = | yet (3.222) 
Then s 
RQ =in® niol=[P ES | (3.223) 
and 
det[R(t)] =1+e' forallt (3.224) 


where “det” denotes the determinant. Since det[R(t)] 4 0 for all t, R(t) has rank 2 for all t, and thus the 
control canonical form exists for the system given by Equations 3.211 and 3.212. 

The rows c;(t) of the matrix P~!(t) in the transformation z(t) = P~!(t)x(t) to the control canonical 
form are computed using Equations 3.127 and 3.128. This yields 


c(t)=[—( +e)! 0] and o(t)=[-e “(+e ")? 1 (3.225) 
and thus Mets ; 
Spi) ROGER) = 0 a —(1+e) 0 
P Ce nee and Ce ore (3.226) 


Now choose the coefficients of the characteristic polynomial of the closed-loop system matrix in the 
control canonical form to be dy and d,. Then from Equation 3.136, the feedback gain vector g(t) back in 
the original state coordinates is given by 


g(t) = — (c3(t) + doci(t) + dyco(t)) (3.227) 

where 
c3(t) = co(t)A(t) + c(t) (3.228) 
aQat2e ihe Ye te I eee Se (3.229) 


Inserting the expressions for c(t), c2(f), and c3(t) given by Equations 3.225 and 3.229 into Equation 3.227 
results in the feedback gain vector g(t). 

Since P(t) and P~!(t) given by Equation 3.226 are bounded for t > t; for any finite t), by choosing 
appropriate values for dy and d, it follows that via the state feedback control given by u(t) = —g(t)x(t), 
the resulting closed-loop system is exponentially stable with any desired rate ) of convergence to zero. It 
also follows that with the state feedback control u(t) = —g(t)x(t), where X(t) is the estimated state, the 
resulting closed-loop system is also exponentially stable with any desired rate of convergence to zero. 
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3.4.2 Exponential Systems 


A system with n-dimensional state model [A(t), B(t), C(t)] is said to be an exponential system if its state- 
transition matrix ®(t, t) can be written in the matrix exponential form 


(t,t) =e 9 (3.230) 


where I(t, t) is a m x n matrix function of t and t. The form given in Equation 3.230 is valid (at least 
locally, that is, when t is close to t) for a large class of time-varying systems. In fact, as noted above, for 
t € [t,t +T], P(t, t) can be approximated by the matrix exponential e44"('— if A(t) is approximately 
equal to A(t) for f € [t, t+ T]. For a mathematical development of the matrix exponential form, see [5]. 

As noted previously, the exponential form in Equation 3.230 is valid for any system where A(f) 
commutes with its integral (see Equation 3.47), in which case 


T(t, t) = [4 do (3.231) 


The class of systems for which A(t) commutes with its integral is actually fairly large; in particular, this is 
the case for any A(t) given by 


A(t) = So fii (3.232) 


i=1 


where fj(t) are arbitrary real-valued functions of t and the A; are arbitrary constant n x n matrices that 
satisfy the commutativity conditions 


AjAj = AjAi; for all integers 1 <i,j <r (3.233) 
For example, suppose that 
fil) a 
A(t) = 3.234 
so Be fill) Ge) 


where f(t) and f2(t) are arbitrary real-valued functions of t and c and cz are arbitrary constants. Then 


A(t) =fi(Ai + fr(A2 (3.235) 
where A, =I and 
Age : a (3.236) 
C2 0 


Obviously, A; and Az commute, and thus ®(t, t) is given by Equations 3.230 and 3.231. In this case, 
®@(t, t) can be written in the form 


t t 
@(t, t) = exp If filo) ao) i exp If foo) ao) Ar] (3.237) 


Given an n-dimensional system with exponential state-transition matrix P(t, t) = e! (60) @(t,t) can 
be expressed in terms of scalar functions using the Laplace transform as in the time-invariant case. 
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In particular, let 
®(t,B, t) = inverse transform of [sI — (1/B)I'(p, yy (3.238) 


where I'(f, t) = I(t, t)|+=g and B is viewed as a parameter. Then 


(t,t) = B(E,B, T)lg—y (3950) 
For example, suppose that 
{AM ft) 
ayes Ee | (3.240) 


where f(t) and f2(¢) are arbitrary functions of t with the constraint that f)(t) > 0 for all t. Then 


(t,t) = [4 do (3.241) 


and ®(t, B, t) is equal to the inverse transform of 
-1 
s—yi(B,t) —ya(B, t) 
P(s, 8, t) = 3.242 
ae yB.t)—s—yi(60) a 


where 
B 
vi(8.2) = (1/8) / flyae 5a) 


B 
y2(B, 0) = (1/8) i filo) do Gaan 


Computing the inverse Laplace transform of ®(s, 8, t) and using Equation 3.239 give (for t > 0) 


(3.245) 


yi(t,t)t y1(t,t)t 
Seat | e cos[y2(t, t)t] e sin[y2(t, S| 


—eMEYsinfy(t, Dt] eV! cos[y2(t, DE] 


3.4.3 Stability 


Again consider an n-dimensional exponential system [A(t), B(t),C(t)] with state-transition matrix 
(t,t) =e. A sufficient condition for exponential stability of the differential equation x(t) = 
A(t)x(t)is that the n x n matrix (1/t)I'(t, t) be bounded as a function of t and its pointwise eigen- 
values have real parts < —v for some v > 0 and all t > t for some finite t. For example, suppose that A(t) 
is given by Equation 3.234 so that 


yilt,t) — crya(t, | 
1/t)T(t, t) = 3.246 
COE!) Ba t) = -yi(t,t) ( 
where y(t, t) and y2(t, t) are given by Equations 3.243 and 3.244 with B = t. Then 
det[sI — 1/HT(t, 1] = ve 2yi(t, ts + vilt, tO c1eya(t, Tt) (3.247) 


and the pointwise eigenvalues of (1/t)I'(t, t) have real parts < —v for all t > t for some v > 0 and tif 


yilt,t)<v, forallt>t and somev, <0, (3.248) 
vilt, Tt c1cryx(t, t)>v2 forallt>t and some v.>0 (3.249) 


Therefore, if Equations 3.248 and 3.249 are satisfied and y;(t, t) and y2(t, t) are bounded functions of t, 
the solutions to x(t) = A(t)x(t) with A(t) given by Equation 3.234 decay to zero exponentially. 
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It is well known that in general there is no pointwise eigenvalue condition on the system matrix A(t) 
that insures exponential stability, or even asymptotic stability. For an example (taken from [8]), suppose 
that 


= —1+a(cos*t) 1 — a(sin t)(cos t) 
AS he — a(sin t)(cos t) —1+a(sin?t) (3.280) 
where @ is a real parameter. The pointwise eigenvalues of A(t) are equal to 
—24+V0*—-4 
coher eM (3.251) 
2 
which are strictly negative if 0 < a < 2. But 
(a-1t( —tle 
e cost) e‘(sint) 
P(t,0) = 5 3.252 
(¢,0) ee t) e (cos | ( ) 


and thus, the system is obviously not asymptotically stable if a > 1. 


3.4.4 The Lyapunov Criterion 
By using the Lyapunov criterion (see Equation 3.97), it is possible to derive sufficient conditions for 


uniform exponential stability without computing the state-transition matrix. For example, suppose that 


0 1 
A(t) = E sO al (3.253) 


where a(t) is a real-valued function of t with a(t) > c for all t > t), for some t; and some constant c > 0. 
Now in Equation 3.97, choose 


2 
a(t) ++ —~ 
Q(t) = a os (3.254) 
a(t) 
Then, c)I < Q(t) < c, for all t > t; for some constants c, > 0 and cz > 0. Now 
a(t 
~2+a(t)- 7 
QA) + AN (HAL) + QU) = ° a(t) (3.255) 
0 —-l- 
a’(t) 
Hence, if 
: a(t) 
2+ a(t) <-c3 fort>t, forsomec3 >0 (3.256) 
a*(t) 
and 
a(t) 
—1——— <-cq, fort>t, forsomec,>0 (3.257) 
a*(t) 


the system is uniformly exponentially stable. For instance, if a(t) = b — cost, then Equations 3.256 and 
3.257 are satisfied if b > 2, in which case the system is uniformly exponentially stable. 

Now suppose that 

0 1 
A(t) = 3.258 
O-[ ao —a(| we 

As suggested in [8, p. 109], sufficient conditions for uniform exponential stability can be derived by 

taking 


Pe rae. 


Q(t) = aa(t) i (3.259) 
l + 
a(t) 
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3.5 Defining Terms 


State model: For linear time-varying systems, this is a mathematical representation of the system in terms 
of state equations of the form x(t) = A(t)x(t) + B(t)u(t), y(t) = C(t)x(t) + D(t)u(t). 

State-transition matrix: The matrix ®(t, to) where P(t, to)x(to) is the state at time ¢ starting with state 
x(t) at time f and with no input applied for t > fo. 

Exponential system: A system whose state-transition matrix ®(f, t) can be written in the exponential 
form e! +” for some n x n matrix function I(t, t). 

Reversible system: A system whose state-transition matrix is invertible. 

Sampled data system: A discrete-time system generated by sampling the inputs and outputs of a 
continuous-time system. 

Change of state: A transformation z(t) = P~1(t)x(t) from the state vector x(t) to the new state vector 
z(t). 

Algebraic equivalence: Refers to two state models of the same system related by a change of state. 

Lyapunov transformation: A change of state z(t) = P~1(t)x(t) where P(t) and its inverse P~!(t) are both 
bounded functions of t. 

Canonical form: A state model [A(t), B(t),C(t)] with one or more of the coefficient matrices 
A(t), B(t), C(t) in a special form. 

Control canonical form: In the single-input case, a canonical form for A(t) and B(t) that facilitates the 
study of state feedback control. 

Observer canonical form: In the single-output case, a canonical form for A(t) and C(t) that facilitates 
the design of a state observer. 

Characteristic vector: A time-varying generalization corresponding to the vector of coefficients of the 
characteristic polynomial in the time-invariant case. 

Asymptotic stability: Convergence of the solutions of x(t) = A(t)x(t) to zero for any initial state x(t). 

Exponential stability: Convergence of the solutions of x(t) = A(t)x(t) to zero at an exponential rate. 

Uniform exponential stability: Convergence of the solutions of x(t) = A(t)x(t) to zero at an exponential 
rate uniformly with respect to the initial time. 

Pointwise eigenvalues: The eigenvalues of a n x n time-varying matrix M(t) with t replaced by t, where 
t is viewed as a time-independent parameter. 

Controllability: The existence of inputs that drive a system from any initial state to any desired state. 

Observability: The ability to compute the initial state x(to) from knowledge of the output response y(t) 
for t > to. 

State feedback control: A control signal of the form u(t) = —F(t)x(t) where F(t) is the feedback gain 
matrix and x(t) is the system state. 

Observer: A system which provides an estimate x(t) of the state x(t) of a system. 


Acknowledgment 


The author wishes to thank Professor Wilson J. Rugh of the Johns Hopkins University for his comments 
regarding a technical issue involving the concept of uniform exponential stability in the time-varying 
case. 


References 


1. Chen, C. T., Linear System Theory and Design, Holt, Rinehart, and Winston, New York, 1984. 
2. Kamen, E.W., The poles and zeros of a linear time-varying system, Linear Algebra Appl., 98, 263-289, 
1988. 


3-32 Control System Advanced Methods 


3. Kamen, E.W. and Hafez, K.M., Algebraic theory of linear time-varying systems, SIAM J Control Optim , 
17, 500-510, 1979. 
4. Kloet, P. van der and Neerhoff, F.L., The Cauchy—Floquet factorization by successive Riccati transfor- 
mations, Proc. IEEE Int. Sym. Circuits Systems, Phoenix, AZ, 257-260, 2002. 
5. Magnus, W., On the exponential solution of differential equations for a linear operator, Commun. Pure 
Appl. Math., VII, 649-673, 1954. 
6. Marinescu, B. and Bourlés, H., An intrinsic algebraic setting for poles and zeros of linear time-varying 
systems, Systems Control Lett., 58, 248-253, 2009. 
7. O’Brien, R.T. and Iglesias, P. A., On the poles and zeros of linear, time-varying systems, LEEE Trans. 
Circuits Systems-I;: Fundam, Theory Appl., 48, 565-577, 2001. 
8. Rugh, W.J., Linear System Theory, 2nd edn, Prentice-Hall, Englewood Cliffs, NJ, 1996. 
9. Silverman, L.M. and Meadows, H.E., Controllability and observability in time-variable linear systems, 
SIAM J. Control Optim., 5, 64-73, 1967. 
10. Silverman, L.M., Transformation of time-variable systems to canonical (Phase-variable) form, [EEE 
Trans. Automat, Control, AC-11, 300, 1966. 
11. Valasek, M. and Olgac, N., An efficient pole placement technique for linear time-variant SISO systems, 
IEE Proc. Control Theory Appl. D, 451-458, 1995. 
12. Valasek, M. and Olgac, N., Pole placement for linear time-varying non-lexicographically fixed MIMO 
systems, Automatica, 35, 101-108, 1999. 
13. Zhu, J.J., Well-defined series and parallel d-spectra for linear time-varying systems, Proc. Am. Control 
Conf., Baltimore, MD, 734-738,1994. 


Further Reading 


There are a large number of research papers and textbooks that contain results on the theory of linear 
time-varying systems. Only a small portion of the existing work is mentioned here, with the emphasis on 
textbooks: In addition to [1] and [8] listed in the references above, textbooks that contain material on the 
fundamentals of linear time-varying systems include the following: 


14. Zadeh, L.A. and Desoer, C.A., Linear System Theory, McGraw-Hill, New York, 1963. 
15. Brockett, R.W., Finite Dimensional Linear Systems, Wiley & Sons, New York, 1970. 
16. D’Angelo, H., Linear Time-Varying Systems, Allyn and Bacon, Boston, MA, 1970. 
17. Kailath, T., Linear Systems, Prentice-Hall, Englewood Cliffs, NJ, 1980. 

18. Sontag, E.D., Mathematical Control Theory, Springer-Verlag, New York, 1990. 

19. Antsaklis, P.J. and Michel, A.N., Linear Systems, Birkhauser, Boston, MA, 2006. 


For textbooks on H-infinity and H2 control of time-varying systems, see 


20. Peters, M.A. and Iglesias, P.A., Minimum Entropy Control for Time-Varying Systems, Birkhauser, Boston, 
MA, 1997. 

21. Ichikawa, A. and Katayama, H., Linear Time-Varying and Sampled-Data Systems, Lecture Notes in 
Control and Information Science, 265, Springer-Verlag, London, 2001. 


An approach to linear time-varying systems given in terms of matrix algebra and analytic function theory 
is developed in the following textbook: 


22. Dewilde, P. and Veen, A. van der, Time-Varying Systems and Computations, Kluwer, Boston, MA, 1998. 


For textbooks on the adaptive control of time-varying systems and observers for time-varying systems, 
see 


23. Tsakalis, K.S. and Ioannou, P.A., Linear Time-Varying Plants: Control and Adaptation, Prentice-Hall, 
Englewood Cliffs, NJ, 1993. 
24. O'Reilly, J., Observers for Linear Systems, Academic Press, New York, 1983. 


Many textbooks exist on the stability of time-varying differential equations and systems. Examples are 


25. Bellman, R., Stability of Differential Equations, McGraw-Hill, New York, 1953. 

26. Hahn, W., Stability of Motion, Springer-Verlag, New York, 1967. 

27. Harris, C.J. and Miles, J.F., Stability of Linear Systems, Academic Press, New York, 1980. 

28. Lukes, D.L., Differential Equations: Classical to Controlled, Academic Press, New York, 1982. 

29. Miller, RK. and Michel, A.N., Ordinary Differential Equations, Academic Press, New York, 1982. 


Fundamentals of Linear Time-Varying Systems 3-33 


30. Michel, A.N., Liu, D., and Hou, L., Stability of Dynamical Systems: Continuous, Discontinuous, and 
Discrete Systems, Birkhauser Verlag, Boston, MA, 2008. 


For an in-depth treatment of the stability of second-order linear time-varying differential equations, see 


31. Duc, L.H., Ilchmann, A., Siegmund, S., and Taraba, P., On stability of linear time-varying second-order 
differential equations, Quart. Appl. Math. 64, 137-151, 2006. 


There are a number of papers on the study of time-varying systems given in terms of rings of differential 
or difference polynomials. In addition to [2], [3], and [6] in the references above, examples of this work 
are as follows: 


32. Kamen, E.W., Khargonekar, P.P., and Poolla, K.R., A transfer function approach to linear time-varying 
discrete-time systems, SIAM J. Control Optim., 23, 550-565, 1985. 

33. Poolla, K. R. and Khargonekar, P.P., Stabilizability and stable proper factorizations for linear time- 
varying systems, SIAM J. Control Optim., 25, 723-736, 1987. 

34. Fliess, M., Some basic structural properties of generalized linear systems, Systems Control Lett. 15, 
391-396, 1990. 


Balanced Realizations, 
Model Order Reduction, 
and the Hankel Operator 


20 OCCT ccnamnmuosiamnanamned 
4.2 Linear Systems 
The Hankel, Controllability, and Observability 
Operator + Balanced State-Space Realizations « 
Model Reduction + Unstable Systems, 
Closed-Loop Balancing 
4.3 Balancing for Nonlinear Systems............00000 4-13 
Basics of Nonlinear Balanced Realizations « 
Balanced Realizations Based on Singular- Value 
Analysis of Hankel Operators « Model Order 
Reduction « Other Types of Balancing for 
Nonlinear Systems 


Jacquelien M.A. Scherpen 44 Concluding Remarks oc eeeeeeeeeeeeneees 4-21 
University of Groningen PCC ceeds ica ce ae adeactivd eee aoe 4-22 


4.1 Introduction 


In many engineering applications, processes are described by increasingly complex models that are 
difficult to analyze and difficult to control. Reduction of the order of the model may overcome some 
of these difficulties, but it is quite possible that model reduction incurs a significant loss of accuracy. 
Therefore, the system has to be analyzed in a manner that is useful for the application purpose. Sim- 
plification of the model based on this analysis usually results in a model of lower complexity which is 
easier to handle, and in a corresponding simplification of synthesis procedures for control and filtering 
problems. Furthermore, the simplification decreases the computational effort. Every application has its 
own demands, and different model reduction methods have different properties. 

In [1] two types of approximation methods for large-scale linear systems are discussed, namely 
singular-value decomposition (SVD) and Krylov methods. The Krylov methods are based on match- 
ing the moments of the impulse response of the linear system up to a certain level. This can be interpreted 
in terms of the series expansions of the transfer function; see, for example, [1] and the references therein. 
The key to the success of these methods is that they can be implemented iteratively and are able to handle 
much higher-order systems than the SVD methods. Nevertheless, they are not always resulting in models 
that are useful for control, for example, stability may not be preserved, and no a priori error bounds for 
the reduced order system can be given. Furthermore, an extension of these methods to nonlinear systems 
is still largely unexplored. Only recently, the first steps toward such extension is presented by [2]. The 
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SVD methods based on balanced realizations, on the other hand, offer a clear structure for analysis of the 
system based on controllability and observability properties, and model reduction by balanced truncation 
does preserve stability and other properties and does have an a priori error bound for the reduced order 
system. Furthermore, the extension of balancing to nonlinear systems has been studied in the past two 
decades as well. 

Kalman’s minimal realization theory (e.g., [27]) offers a clear picture of the structure of linear systems. 
However, the accompanying algorithms are not very satisfactory, since they are only textbook algorithms, 
which are numerically deficient. Moore [32] showed that there are very useful tools that may be used to 
cope with this problem. He used the principal component analysis which was introduced in statistics in 
the 1930s to analyze a linear system, and specifically to apply it to model reduction. The most important 
contribution of [32] is the introduction of balancing for stable minimal linear systems. The balancing 
method offers a tool to measure the contribution of the different state components to the past input and 
future output energy of the system, which are measures of controllability and observability. The algo- 
rithmic methods corresponding to the balancing theory nowadays are standard toolboxes in simulation 
packages like MATLAB®, 

In the theory of continuous-time linear systems, the system Hankel operator plays an important role 
in a number of realization problems. For example, when viewed as a mapping from past inputs to future 
outputs, it plays a direct role in the abstract definition of state. It also plays a central role in minimality 
theory, in model reduction problems, and related to these, in linear identification methods. Specifically, 
the Hankel operator supplies a set of similarity invariants, the so-called Hankel singular values, which can 
be used to quantify the importance of each state in the corresponding input-output system. The Hankel 
operator can also be factored into the composition of an observability and controllability operator, from 
which Gramian matrices can be defined and the notion of a balanced realization follows. The Hankel 
singular values are most easily computed in a state-space setting using the product of the Gramian 
matrices, though intrinsically they depend only on the given input-output mapping. For linear systems, 
the Hankel operator offers an immediate relation with the frequency domain setting of balancing for 
linear systems, for example, [63]. 

Furthermore, these methods have proved to be very useful for application purposes. To mention a 
few older applications, we refer to [16,62]. Reference [62] successfully applies methods based on bal- 
anced realizations on the controller design of the Philips CD player. In [16], several balanced controller 
designs and algorithms for sensor and actuator placement based on balanced realizations are given and 
demonstrated for the NASA Deep Space Network Antenna. 

For nonlinear systems, the first step toward extension of the linear balancing methods has been set 
in [45], where a balancing formalism is developed for stable nonlinear continuous-time state-space sys- 
tems based on the idea that state components that correspond to low control costs and high output energy 
generation (in the sense of Lz energy in a signal) are important for the description of the dynamics of 
the input-output behavior of the system, while state components with high control costs and low out- 
put energy generation can be left out of the description. Since then, many results on state-space balancing, 
modifications based on sliding time windows, and modifications based on proper orthogonal decompo- 
sition (POD), and computational issues for model reduction and related minimality considerations for 
nonlinear systems have appeared in the literature; for example, [18,20,28,37,38,58,59,64]. The relations 
of the original nonlinear balancing method of [45] with minimality are later explored in [47], and a 
more constructive approach that includes a strong relation with the nonlinear input-output operators, 
the nonlinear Hankel operator, and the Hankel norm of the system is presented in [10,12,48]. Model 
reduction based on these studies is recently treated in [13,14]. 

Here we first review the Hankel operator, balancing and balanced truncation for stable linear systems. 
Then unstable systems and closed-loop balancing are reviewed for linear systems. Finally, the extension 
of the linear theory to nonlinear systems is treated, based on balancing procedures in a neighborhood 
(possibly large, almost global) of an equilibrium point. 
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4.2 Linear Systems 


In this section, we briefly review the well-known linear system definitions of the system Hankel matrix; 
the Hankel operator; the controllability and observability operators, Gramians and functions; and the 
balanced realization and the corresponding model reduction procedure. 


4.2.1 The Hankel, Controllability, and Observability Operator 


Consider a continuous-time, causal linear input-output system : u— y, with m-dimensional input 
space u € U and p-dimensional output space y € Y, and with impulse response H(t). Let 


ee (4.1) 


denote its Taylor series expansion about t = 0, where Hy € RP*” for each k. The system Hankel matrix 
is defined as H = [Hi jl, where Hij = Hj4j-1 for i,j = 1. If & is also (bounded input bounded output) 
BIBO stable, then the system Hankel operator is the well-defined mapping 


H. : L%'[0, +00) > LE[0, +00), 


[ua p(t) = ii H(t + 1)u(t) dt. (4.2) 
0 


mapping the past inputs to the future outputs. If we define the time flipping operator as 
F: L"[0, +00) > L5"(—00, 0] 
i(—t):t <0 
[us uo={% yes 

0:t>0, 
then clearly H(a) = (XoF) (i); see the illustration in Figure 4.1. The lower side of the figure depicts the 
input-output behavior of the original operator &. The upper side depicts the input-output behavior of 
the Hankel operator of X, where the signal in the upper left side is the time-flipped signal of the lower 
left side signal. The flipping operator F is defined by F(u(t)) := u(—t). The upper right side signal is 
the truncated signal (to the space L2[0,00)) of the lower left side signal. The corresponding truncation 
operator is given by 
0 (t<0) 


TONS ie (t>0)° 


and F = T o F. The definition of a Hankel operator implies that it describes the mapping from the input 
to the output generated by the state at f = 0. Hence, we can analyze the relationship between the state and 
the input-output behavior of the original operator X by investigating its Hankel operator. 

When H is known to be a compact operator, then its (Hilbert) adjoint operator, H*, is also compact, 
and the composition H*H, is a self-adjoint compact operator with a well-defined spectral decomposition: 


[o,2) 
HH=) of Wn Wi 9720, (4.3) 
i=l 
(Wis Vin =i (Wi (HHA) 1, = 07, (4.4) 
where o? is an eigenvalue of H*H with the corresponding eigenvector wy;, ordered as 0) > --- >On > 


Ont+1 = On+2 = 0, and called the Hankel singular values for the input-output system ©. 
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as 


FIGURE 4.1 Hankel operator H of X. 


Let (A, B, C) be a state-space realization of & with dimension n, that is, consider a linear system: 


x= Ax+ Bu, (4.5) 
y=Cx, 


where u € IR”, x € R", and y € R’. We assume that Equation 4.5 is stable and minimal, that is, control- 
lable and observable. Any such realization induces a factorization of the system Hankel matrix into the 
form H = OC, where © and @ are the (extended) observability and controllability matrices. If the real- 
ization is asymptotically stable, then the Hankel operator can be written as the composition of uniquely 
determined observability and controllability operators; that is, = OC, where the controllability and 
observability operators are defined as 


lo) 
C: Lt'[0, +00) > R" as i e“ Balt) dt, 
0 


o:R’s Ifo, +00): x > p(t) = CeAty, 


Since C and O have a finite dimensional range and domain, respectively, they are compact operators; and 
the composition OC is also a compact operator. From the definition of the (Hilbert) adjoint operator, it 
is easily shown that C and O have corresponding adjoints 


c* : IR" > L®[0, +00) :x > BleA'ty, 


oO, 
O* : I5[0, +00) > R": y > | eA"tcT y(t) dt. 
0 
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4.2.2 Balanced State-Space Realizations 


The above input-output setting can be related with the well-known Gramians that are related to the state- 
space realization. In order to do so, we consider the energy functions given in the following definition. 


Definition 4.1: 


The controllability and observability functions of a smooth state-space system are defined as 


1 0 
L,(xo) = min = 7 \| u(t) ||? dt (4.6) 
u€L2(—00,0) 2 —0o 
x(—00)=0,x(0)=xo 
and 
1 f®% 2 
Lo(xo) = > ll y@) | dt, x(0)=x, u(t)=0, 0<t<ov, (4.7) 
0 
respectively. 


The value of the controllability function at xo is the minimum amount of input energy required to 
reach the state x9 from the zero state, and the value of the observability function at x is the amount of 
output energy generated by the state xo. The following results are well known: 


Theorem 4.1: 


Consider the system (Equation 4.5). Then L,(xo) = 5x4 P-1xo and Lo(xo) = 5x4 Qxo, where P= 
i e“tBBT eA dt is the controllability Gramian and Q = Ion e4'tCT CeAt dt is the observability Gramian. 
Furthermore, P and Q are symmetric and positive definite, and are unique solutions of the Lyapunov 
equations 

AP + PA! = —BB! (4.8) 


and 
A'Q+QA=—C'C, (4.9) 


respectively. 


From the form of the Gramians in this theorem, it follows immediately that for any x,,x) € R", 
[o,@) T 
<X1,CC*x, > = af f ec’ ' BB e4 ‘dt xp, 
0 
221E 
=x; Px, (4.10) 
a T 
<x1,0*0x. >= af f e 'Cl Cet dt x, 
0 


=x! Quy (4.11) 


and the relation with the energy functions is given as 


L-(x) = re x= 5s (cc*)~!x), (4.12) 
L(x) = paler = 5 bs (O*O)x). (4.13) 


The following (balancing) theorem is originally due to [32]. 


4-6 Control System Advanced Methods 


Theorem 4.2: [32] 


The eigenvalues of QP are similarity invariants, that is, they do not depend on the choice of the sate-space 
coordinates. There exists a state-space representation where 


O1 0 
Y:=Q=P= ; (4.14) 


with 0) > 02 >-+- >On >0 the square roots of the eigenvalues of QP. Such representations are called 
balanced, and the system is in balanced form. Furthermore, the o;’s,i=1,...,n, equal the Hankel singular 
values, that is, the singular values of the Hankel operator (Equation 4.2). 


Two other representations that may be obtained from Equation 4.14 by coordinate transformations 


x= Dox andx = D2Kx, respectively, follow easily from the above theorem. 


Definition 4.2: [32] 


A state-space representation is an input-normal/output-diagonal representation if P=I and Q= ~’, 
where & is given by Equation 4.14. Furthermore, it is an output-normal/input-diagonal representation if 
P=" andQ=I1. 


The largest Hankel singular value is equal to the Hankel norm of the system, that is, 


T Ty2 
x x° Qx x Lex 
lI G (Fs = max ol ) = max aNee = max xT. = Gs (4.15) 
xe Lf ) xe R" x PY'x XE, Re *% 
x40 x40 x40 


where G = C(sI — A)~!B is the transfer matrix of the system. This gives a characterization of the largest 
Hankel singular value. The other Hankel singular values may be characterized inductively in a similar 
way; we refer to [6,15]. 

So far, we have assumed the state-space representation to be minimal. However, if we consider non- 
minimal state-space realizations, that is, the system is not controllable and/or observable, then we obtain 
os that are zero, corresponding to the noncontrollable or nonobservable part of the system, and thus, to 
the nonminimal part of the system. Related to this observation, we have that the minimal realization of 
a linear input-output system has a dimension n that is equal to the Hankel rank, or in other words, it 
equals the rank of the Hankel matrix. A well-known result related to the latter is the following theorem 
for example, [63]. 


Theorem 4.3: 


If (A, B, C) is asymptotically stable, then the realization is minimal if and only if P > 0 and Q > 0. 


4.2.3 Model Reduction 


Once the state-space system is in balanced form, an order reduction procedure based on this form may be 
applied. Thus, in order to proceed, we assume that the system (Equation 4.5) is in balanced form. Then 


Balanced Realizations 4-7 


the controllability and observability function are L. (Xo) = 5xg Xx and L.(X) = 5Xq “Xo, respectively. 
For small o;, the amount of control energy required to reach the state x = (0,...,0,xj,0,...,0) is large 
while the output energy generated by this state x is small. Hence, if 0, >> 0441, the state components x, +1 
to x, are far less important from this energy point of view and may be removed to reduce the number of 
state components of the model. We partition the system (Equation 4.5) in a corresponding way as follows: 


Ai ‘ |) 
ies . He 3G H(Ce Es 4.16 
ee A22 By ( : 2) le) 
x 0 
x = (x1,...,x4))5 x? = (Kept. .-oXn))s “= ( x ah 
where X) = diag(o1,...,0,%) and Xz = diag(ox41,..-,0n). 


Theorem 4.4: 


Both subsystems (Aji, Bj, C;), i = 1,2, are again in balanced form, and their controllability and observability 
Gramians are equal to Xj, i = 1,2. 


The following result has been proved by [43]. 


Theorem 4.5: 


Assume that 04 > 0441. Then both subsystems (Ajj, Bj, C;), i= 1,2, are asymptotically stable. 


The subsystem (Aj), Bj, C;) may be used as an approximation of the full order system (Equation 4.5). 
The optimality of this approximation in the Hankel and H.o-norm has been studied by [17], and an upper 
bound for the error is given. The Hoo-norm of G(s) = C(sI — A)~'B is defined as 


1 
|Glloo = sup Mhax(G(—jo)" GGw)), 


we 


1 
where Mrax(G(—jw)! G(jo)) is the square root of the maximum eigenvalue of G(—jw)? GGjw). Denote 
the transfer matrix of the reduced order system (A11, Bi, Ci) by G(s) = C1(sI — Ay1)~!B,. The following 
error bound was originally proved in [17]. A number of proofs can be found in [63]. 


Theorem 4.6: [17] 


|G — Gllit < IG — Glloo S UoKp1 + +++ + On). 

Hence, if we remove the state components x;41,. . . ,X, that correspond to small Hankel singular values 
Ok41>+++>On (small compared to the rest of the singular values, that is, o, >> 0,41), then the error is small, 
and the reduced order system (Aj, By, C)) constitutes a good approximation in terms of the Hankel norm 
to the full order system. 

The model reduction method that we gave above consists of simply truncating the model. It is 
also possible to reduce the model in a different way see, for example, [8,21]. Instead of setting 
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x” = (xp41,---»Xn) =0 we approximate the system by setting x? = 0 (thus interpreting x? as a very 
fast stable state, which may be approximated by a constant function of x! and u). The resulting algebraic 
equation can be solved for x? as (note that Ady. exists by Theorem 4.5) 


r= —A>} (Azix! + Bu) é 
Substitution in Equation 4.5 leads to a reduced order model (A, B, C) defined as 


A:=Au— Ay2A5y Ari, 


The system (A, B, C) also gives an approximation to the full order system (Equation 4.5). Theorems 4.4 
through 4.6 also hold if we replace the system (A11, Bi, C1) by (A, B, C). 


4.2.4 Unstable Systems, Closed-Loop Balancing 


4.2.4.1 Linear Quadratic Gaussian Balancing 


A major drawback of the original balancing method as described in Section 4.2.2 is that it only applies 
to stable systems. Furthermore, the method emphasizes the (open-loop) input-output characteristics of 
the system, while it is a priori not clear if it yields good approximations in closed-loop configurations. In 
this section, we treat (linear quadratic Gaussian) LQG balancing for linear systems, which was introduced 
by [25,26] (see also [57]). In [41], this concept is further developed. LQG balancing was introduced 
with the aim of finding a model reduction method for a system (not necessarily stable) together with its 
corresponding LQG compensator. LQG balancing has been treated from another point of view in [61]. 
First, we give a review of the formulation of [26,41]. 


LQG compensation is formulated for a minimal state-space system 


x= Ax+Bu+ Bd, (4.17) 
y=Cx+y, 


where ue IR”, xe R", ye R?, and d and v are independent Gaussian white-noise processes with 
covariance functions I3(t — t). The criterion 


T 
I(x, u(-)) = Elim / [x? (t)C? Cx(t) + u" (t)u(t)] dt (4.18) 
> 0o 0 


is required to be minimized. The resulting optimal compensator is given by 


z=Az+Bu+SCl(y— Cz), (4.19) 
u=—B' Pz. 


Here S is the stabilizing solution (ie, o(A—SC™C) CC ) to the filter algebraic Riccati equation 
(FARE) 
AS + SAT + BB’ —SCTCS =0, (4.20) 


and P is the stabilizing solution (ie., o(A —BB'P) CC ) to the control algebraic Riccati equation 
(CARE) 
A’P+PA+C'C— PBB'P=0. (4.21) 
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Theorem 4.7: [26,41] 


The eigenvalues of PS are similarity invariants and there exists a state-space representation where 


M1 0 
M:=P=S= te : (4.22) 
0 Ln 


with 1 > 2 > +++ > ln > 0. This is called an LQG balanced representation or LQG balanced form. 


Jonckheere and Silverman [26] and Opdenacker and Jonckheere [41] argue that if rp >> x4) then 
the state components x; up to x, are more difficult both to control and to filter than x,4) up to x, and 
a synthesis based only on the state components x; up to x, probably retains the essential properties of 
the system in a closed-loop configuration. Corresponding to the partitioning of the state in the first k 
components and the last n — k components, the partitioning of the matrices is done as in (4.16), and the 
reduced order system is 


x= Ayx+ But Bid, (4.23) 
yH=Cxt+yv. 


Theorem 4.8: [26,41] 


Assume |tk > (te41. Then (A11, Bi, C1) is minimal, the reduced order system (Equation 4.23) is again LQG 
balanced and the optimal compensator for system (Equation 4.23) is the reduced order optimal compensator 
of the full order system (Equation 4.17). 


As explained in Section 4.2.2, the original idea of balancing stable linear systems, as introduced by [32], 
considers the Hankel singular values 0;, i= 1,...,n, which are a measure for the importance of a state 
component in a balanced representation. This balancing technique is based on the input energy which is 
necessary to reach this state component and the output energy which is generated by this state component. 
A similar kind of reasoning, using a different pair of energy functions, may be used to achieve the similarity 
invariants tj, i= 1,...,n, as above; see [61]. To follow this reasoning, we consider the minimal system 
(Equation 4.17) without noise, that is, 


x= Ax+ Bu, (4.24) 
y=Cx, 


where u € IR”, x € R", andy € R’. We define the energy functions 


= a 
Cees. te 2 i (|| y(¢) ||? +l] u(t) IP) dt, 
u€L2(—00,0) 2 Joo 
x(—0o)=0,x(0)=xo 
bf 2 2 
Cee ain. 2 / (I) y(t) |? + Il wl) [I2) at. 
u€L2(0,00) 2 0 


x(00)=0,x(0)=xo 


K~ (xp) is called the past energy and Kt (xo) the future energy of the system in the state xq. 
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Theorem 4.9: [61] 


K~ (x0) = 5xq 5! x0 and K* (xo) = 5x4 Pxo, where S and P are the stabilizing solutions of the FARE and 
the CARE, Equations 4.20 and 4.21, respectively. 


For the LQG balanced representation of Theorem 4.7, the past and future energy function are K~ (xp) = 
5xG M~!xo and K*(xg) = 5xg Mxo, respectively, where M is diagonal. The importance of the state x = 
(0,...,0,xj,0,...,0) in terms of past and future energy may be measured by the similarity invariant Jj. 
For large \1; the influence of the state x on the future energy is large while the influence on the past energy 
is small. Hence, if |r, >> [Ug41, the state components x,,) to x, are “not important” from this energy 
point of view and may be removed to reduce the number of state components of the model. 


4.2.4.2 Balancing of the Normalized Coprime Representation 


In [29,39], balancing of the normalized coprime representation of a linear system is treated. Balancing of 
the normalized coprime representation was introduced with the aim of finding a model reduction method 
for unstable linear systems. In [29], balancing of the normalized right coprime factorization is treated, 
while in [39], balancing of the normalized left coprime factorization is treated. Here we provide a brief 
review on this subject. 

We consider the system (Equation 4.24) and its transfer function G(s) = C(sI — A)~'B. Furthermore, 
we consider the stabilizing solution P to the CARE (Equation 4.21) and the stabilizing solution S to the 
FARE (Equation 4.20), leading to the stable matrices A := A — BB! P and A := A — SC'C. First we treat 
normalized right coprime factorizations and then normalized left coprime factorizations. 

We may write any transfer matrix G(s) = C(s! — A)~!B asa right fraction G(s) = N(s)D(s)~! of stable 
transfer matrices N(s) and D(s). If we choose (e.g., [36]), 


N(s) := C(sI — A)~1B, 
D(s) :=1—B" P(sl — A)~!B, 


then the factorization is right coprime, that is, N(s) and D(s) have no common zeros at the same place in 
the closed right half plane. A state-space realization of the transfer matrix (the so-called graph operator) 


N(s) 
(503) 


x = (A — BB’ P)x + Bw, 


(:) a (sr) a (7) ", (4.25) 


with w a (fictitious) input variable. Furthermore, we are able to find stable transfer matrices U(s) and 
V(s), such that the Bezout identity 


is 


U(s)N(s) + V(s)D(s) =I. (4.26) 


is fulfilled. Indeed, take U(s) = B' P(sI — A)~!SC™ and V(s) =I +B! P(sI — A)~!B (see, e.g., [36,60]). 
The fact that we are able to find a stable left inverse of the graph operator, that is, we can find the solutions 
U(s) and V(s) to the Bezout identity (Equation 4.26), is equivalent to the factorization being right coprime. 
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Furthermore, the graph operator is inner, that is, 


1 (5) ve = tol 


N(—s)? N(s) + D(—s)? D(s) = I. 


or 


Therefore, the factorization is called normalized. It is easily checked that the observability Gramian of the 
system (Equation 4.25) is P. Denote its controllability Gramian by R. 
In a similar way, we may write the transfer matrix G(s) as a left fraction G(s) = D(s)~!N(s) of stable 
transfer matrices D(s) and N(s). If we choose (e.g., [36]) 
N(s) = C(sI — A) 'B, 
D(s) = C(sI — A)'SCT =I, 


then this is a left factorization. Obviously, ¥(s) = G(s)ii(s) is equivalent with 0 = N(s)ix(s) — D(s)9(s). 
Moreover, a state-space realization of the transfer matrix 


(N(s) D(s)) 
is 
x= (A-—SC'C)x+(B SC") w, 
z=Cx+(0 —I)w. 


-() 


as the input variable, then the dynamics resulting from setting z = 0 in Equation 4.27 is a state-space 
representation of G(s). We are able to find stable transfer matrices such that the Bezout Identity is 
fulfilled, that is, there exist stable transfer matrices U(s) and V(s), such that 


(4.27) 


If we take 


N(s)U(s) + D(s)V(s) =I. (4.28) 


Indeed, we may take U(s) = B' P(sI — A)~!scT and V(s) =I+C(sI — A)~!scT (see, e.g., Vid,Ne). This 
proves that the factorization is left coprime. Furthermore (N (s) D(s)) is co-inner, that is, 


N(s)N(—s)! + D(s)D(—s)? =1, 


which means that the factorization is normalized. Hence (N (s) D(s)) represents the normalized left 
coprime factorization of system (Equation 4.24). The system (Equation 4.27) has as controllability 
Gramian the positive-definite matrix S and we denote its observability Gramian by the matrix Q. Note 


that the right factorization 
N(s) 
D(s) 


can be seen as an image representation of G(s), while the left factorization 
(N(s)_ D(s)) 


can be regarded as a kernel representation of G(s). 
The following result follows rather straightforwardly. 
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Theorem 4.10: 


The Hankel singular values of the right and left factorization (Equations 4.25 and 4.27), respectively, are 
the same. 


Proof. It follows from the Lyapunov Equations 4.8 and 4.9 for the systems (Equations 4.25 and 4.27), 
that R = (I+ SP)~!S and Q= (I+ PS)~!P. Now, it is easily obtained that PR and SQ have the same 
eigenvalues. 


The Hankel singular values of Equation 4.25 (and, hence, of Equation 4.27) are called the graph Hankel 
singular values of the original system (Equation 4.24). These singular values have the following property: 


Theorem 4.11: [29,39] 


The graph Hankel singular values of system (Equation 4.24) are strictly less than one. 


Denote the graph Hankel singular values by t;, i= 1,...,n, and assume 1 >--- > ty. The relation 
between 1;, i=1,...,n, and the similarity invariants 1j;,i=1,...,n, of Theorem 4.7 is given by the 
following theorem: 


Theorem 4.12: [39,61] 


1 
wi=tu(l—v)-2 fori=l,...,n. 


This implies that the reduced model that is obtained by model reduction based on balancing the (left or 
right) normalized coprime factorization is the same as the reduced model that is obtained by model reduc- 
tion based on LQG balancing. Consider the normalized right coprime representation (Equation 4.25) and 
assume that it is in balanced form with 

T] 0 
A:i=P=R= 
0 Hh 


Furthermore, assume that t, > t,+) and define correspondingly A =: diag{Aj, Ag}. It follows, [29], that 
reducing the order of Equation 4.25 by truncating the system to the first k state components (the parti- 
tioning is done corresponding to Equation 4.16) again gives a normalized right coprime representation. 


Theorem 4.13: [29] 


The reduced order system of Equation 4.25 is of the form 


Cy 0 
A ~ B,BEDs) Bi ( eG) 
(( 11 — B, B,D, ), By BT D, I 


with controllability and observability Gramian A,. This system is the normalized right coprime represen- 
tation of the system (Au, Bi, Ci), which is minimal. 
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4.2.4.3 Extensions to Other Types of Balancing 


The methods described above can be put in a more general setting, and extended to the Hoo case. Hoo 
balancing for linear systems is introduced in [33-35]. For details, we refer to the latter references. In the 
Hoo case, balancing is performed on Q,, and Qy , [46], which are defined as 


1 0 
Q) (x)= min = i, (1—y III? + Hu(e)|I? dé, x(—00) = 0, x(0) = x0, Vy 


u€L2(—00,0) 2 —~0o 


and 


I: 2 1 
+ _ z 4) 2: _— — 
QF-G%0) = min > [ IYI + ya dt, x(00)=0, x0) =x, fory > 1, 


while if y < 1, then 


Q) (x0) 


Co 
= max + / IIy()|2-+ ——llu(IP dt, x(00) =0, x(0) = 20. 
u€él2(0,00) 2 Jo l-y 
There is an immediate relation with the solutions to the Hoo Filter and Control Algebraic Riccati equations, 
see, for example, [46]. 
Positive real and bounded real balancing can be done by considering dissipativity with respect to a 
quadratic supply rate that depends on the input and the output of the system: 


1 
s(u, y) = slut yV A , (4.29) 


with ue R”, ye R?, and JE R” HP) P) such that J = J?; see, for example, [23,52]. The bounded 


real balancing case , that is, J = E 2 is treated in [1,22,39,42]. The positive real balancing case, 


0 iL 
that is balancing of strictly passive, asymptotically stable, minimal systems, that is J = | 1 of is treated 


in [1,7,19]. For all these methods it holds that truncation preserves the original balanced structure, and 
thus truncation based on bounded real or positive real balancing preserves bounded and positive realness, 
respectively. Furthermore, error bounds for model reduction based on several of the above mentioned 
methods are available; see, for example, [52]. 

Additionally, model reduction methods based on balancing methods that preserve some internal 
structure of the system currently receive a lot of interest due to applications that require this, such as 
electrical circuit simulators. For some results on Hamiltonian structure preservation, and second-order 
system structure preservation; see, for example, [30,54,55]. 


4.3 Balancing for Nonlinear Systems 


Balanced realizations and the related model order reduction technique rely on singular-value analysis. 
The analysis is important since it extracts the gain structure of the operator, that is, it characterizes the 
largest input-output ratio and the corresponding input [51]. Since linear singular values are defined as 
eigenvalues of the composition of the given operator and its adjoint, it is natural to introduce a nonlinear 
version of adjoint operators to obtain a nonlinear counterpart of a singular value. There has been done 
quite some research on the nonlinear extension of adjoint operators, for example, [3,10,48], and the 
references therein. Here we do not explicitly use these definitions of nonlinear adjoint operators. We rely 
on a characterization of singular values for nonlinear operators based on the gain structure as studied 
in [9]. The balanced realization based on this analysis yields a realization that is based on the singular 
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values of the corresponding Hankel operator, and results in a method which can be viewed as a complete 
extension of the linear methods, both from an input-output and a state-space point of view [12]. 

The related model order reduction technique, nonlinear balanced truncation, preserves several impor- 
tant properties of the original system and the corresponding input-output operator, such as stability, 
controllability, observability, and the gain structure [11,14]. 

This section gives a very brief overview of the series of research on balanced realization and the related 
model order reduction method based on nonlinear singular-value analysis. We refer to [13] for more 
details. 


4.3.1 Basics of Nonlinear Balanced Realizations 


This section gives a nonlinear extension of balanced realization introduced in the Section 4.2. Let us 
consider the following asymptotically stable input-affine nonlinear system 


D: (; =f(x)+g(x)u x0) =x°, (4.30) 


y = h(x), 


where x(t) € R”, u(t) € R”, and y(t) e R?. The controllability operator C: U > X with X = R” and 
U = L3"[0, oo), and the observability operator O : X > Y with Y= re [0, co) for this system are defined 


by 


ee x= —f(x)—g(x)u x(oo)=0 
; “| x9 = x(0) 


O:x by: e =) x(0) = x! 

y = h(x) 
This definition implies that the observability operator O is a map from the initial condition x(0) = x° to 
the output L2 signal when no input is applied. To interpret the meaning of C, let us consider a time-reversal 
behavior of the C operator as 


tists tion anes  g(x)u(—t)_x(—00) = 0 
x9 = x(0) 

Then the controllability operator C can be regarded as a mapping from the input L2 signal to the terminal 

state x(0) = x° when the initial state is x(—oo) = 0. Therefore, as in the linear case, C and O represent the 

input-to-state behavior and the state-to-output behavior, respectively, and the Hankel operator for the 

nonlinear system X in Equation 4.30 is given by the composition of C and O 


H:=Oo0C. (4.31) 


To relate the Hankel, controllability and observability operator with the observability and controllability 
functions defined in Equations 4.6 and 4.7, we first introduce a norm-minimizing inverse C* : X > U of 
C. 


(Ane ne arg min_ |u| 
C(u)=x° 


The operators C' and © yield the definitions of the controllability function L.(x) and the observability 
function L(x) that are generalizations of the controllability and observability Gramians, respectively 
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that is, 


1 1 f° 
LAx c= 5 Co) P= = / || u(t) ||? de, (4.32) 


min 
ueLz(—00,0] 2 
x(—00)=0,x(0)=xo 


Lo(x®) = ; || O(x°) |= ; / LOI dt, x0) =x, ult)=0, O<t<0o. (4.33) 
0 


For linear systems, the relation with the Gramians is given in Theorem 4.1. Here the inverse of P appears 
in the equation for L, because C* appears in the definition (Equation 4.32), whereas C can be used in the 
linear case. In order to obtain functions L,(x) and Lo(x), we need to solve a Hamilton-Jacobi equation 
and a Lyapunov equation. 


Theorem 4.14: [45] 


Consider the system of Equation 4.30. Suppose that 0 is an asymptotically stable equilibrium point and that 
a smooth observability function Lo(x) exists. Then Lo(x) is the unique smooth solution of 
OL g(x) 
Ox 


with Lo(0) = 0. Furthermore, assume that a smooth controllability function L,(x) exists. Then L,(x) is the 
unique smooth solution of 


f(x) + ; h(x)" h(x) = 0, 


AL, (x) 1 AL<(x) paLelx)" _ 


x f(x) 4 war g(x)g(x) Bie , 


with L.(0)=0 such that 0 is an asymptotically stable equilibrium point of x = —f(x)—g(x)g(x)' 
(OL,(x)/Ox)". 


Similar to the linear case, the positive definiteness of the controllability and observability functions 
implies strong reachability and zero-state observability of the system & in Equation 4.30, respectively. 
Combining these two properties, we can obtain the following result on the minimality of the system. 


Theorem 4.15: [47] 


Consider the system of Equation 4.30, and assume it is analytic. Suppose that 


0 < L(x) < co, 
0 < L(x) < 00, 


hold for allx 4 0. Then the system is a minimal realization as defined in and under the conditions from, [24]. 


L(x) and Lo(x) can be used to “measure the minimality” ofa nonlinear dynamical system. Furthermore, 
a basis for nonlinear balanced realizations is obtained as a nonlinear generalization of Definition 4.2 in 
the linear case. For that, a factorization of L(x) into a semiquadratic form needs to be done, that is, in a 
convex neighborhood of the equilibrium point 0, we can write 


OL 
Ox? 


Lo(x) = 5xTM (as, with M(0) = (0). 


Now, an input-normal/output-diagonal form can be obtained. 
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Theorem 4.16: [45] 


Consider the system of Equation 4.30 on a neighborhood W of 0. Suppose that 0 is an asymptotically stable 
equilibrium point, that it is zero-state observable, that smooth controllability and observability functions 
L(x) and L(x) exist on W, and that (67L,/Ox*)(0) > 0 and (87L,/Ax7)(0) > 0 hold. Furthermore, assume 
that the number of distinct eigenvalues of M(x) is constant on W. Then there exists coordinates such that 
the controllability and observability functions L,(x) and Lo(x) satisfy 


1 n 
Le(x) = 5 mie (4.34) 
1a) = 5 Doxa (4.35) 


where (x) = ta(x) = +++ = t(x). 


A state-space realization satisfying the conditions (Equations 4.34 and 4.35) is called an input-normal 
form, and the functions t;(x), i= 1,2,...,n are called singular-value functions. We refer to [45] for the 
construction of the coordinate transformation that brings the system in the form of Theorem 4.16. If 
a singular value function 1;(x) is larger than t;(x), then the coordinate axis x; plays a more important 
role than the coordinate axis x; does. Thus this realization is similar to the linear input-normal/output- 
diagonal realization of Definition 4.2, and it directly yields a tool for model order reduction of a nonlinear 
systems. However, a drawback of the above realization is that the the singular-value functions t;(x)’s 
and consequently, the corresponding realization are not unique, for example, [18]. For example, if the 
observability function is given by 


1 1 
L(x) = 5 (xt (x) + x3T2(x)) — 5 2x1 + x + mwas 
with the state-space x = (x1, x2), then the corresponding singular-value functions are 


T(x) =2+ kx3, 


ta(x) = 14+ (1—K)xf, 


with an arbitrary scalar constant k. This example reveals that the singular value function are not uniquely 
determined by this characterization. To overcome these problems, balanced realizations based on non- 
linear singular value analysis is presented in the following section. 


4.3.2 Balanced Realizations Based on Singular-Value Analysis 
of Hankel Operators 


In this section, application of singular-value analysis to nonlinear Hankel operators determines a balanced 
realization with a direct input-output interpretation, whereas the balanced realization of Theorem 4.16 
is completely determined based on state-space considerations only. To this end, we consider the Hankel 
operator H : U — Y as defined in Equation 4.31 with U = L}"[0, 00) and Y = Ifo, oo). Then a singular- 
value analysis based on the differential form, [12], is given by 


(dH(v))* H(vy)=rv, 2ER, veu, (4.36) 


with } and v the eigenvalues and corresponding eigenvectors, respectively. Since we consider a singular- 
value analysis problem on Lz spaces, we need to find state trajectories of certain Hamiltonian dynamics; 
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see, for example, [12]. In the linear case, we only need to solve an eigenvalue problem on a finite 
dimensional space X = R" to obtain the singular values and singular vectors of the Hankel operator H. 
Here we provide the nonlinear counterpart as follows. 


Theorem 4.17: [12] 


Consider the Hankel operator defined by Equation 4.31. Suppose that the operators C' and O exist and are 
smooth. Moreover, suppose that ». € IR and& € X satisfy the following equation: 


OL,(&) OL, (§) 
3 =k BE neR, €exX. (4.37) 
Then 2 is an eigenvalue of (dH(u)*H(u), and 
v:=Cl(€). (4.38) 


That is, v defined above is a singular vector of H. 


Although the singular-value analysis problem, [13], is a nonlinear problem on an infinite dimensional 
signal space U = L5"[0, 00), the problem to be solved in the above theorem is a nonlinear algebraic 
equation on a finite dimensional space X = IR" which is also related to a nonlinear eigenvalue problem 
on X; see [9]. 

In the linear case, Equation 4.37 reduces to 


where P and Q are the controllability and observability Gramians, and > and & are an eigenvalue and an 
eigenvector of PQ. Furthermore, Equation 4.38 characterizes the relationship between a singular vector 
v of H and an eigenvector & of PQ. Also, in the linear case, there always exist n independent pairs of 
eigenvalues and eigenvectors of PQ. What happens in the nonlinear case? The answer is provided in the 
following theorem. 


Theorem 4.18: [12] 


Consider the system © in Equation 4.30 and the Hankel operator H in Equation 4.31 with X = IR". Suppose 
that the Jacobian linearization of the system has n distinct Hankel singular values. Then Equation 4.37 has 
n independent solution curves & = &;(s), s €¢ R, i=1,2,...,n intersecting to each other at the origin and 
satisfying the condition 


I§i(s) ll = Isl. 


For linear systems, the solutions of Equation 4.37 are lines (orthogonally) intersecting at the origin. 
The above theorem shows that instead of these lines, in the nonlinear case n independent curves x = &;(s), 
i=1,2,...,n exist. For instance, if the dimension of the state is n = 2, the solution of Equation 4.37 is 
illustrated in Figure 4.2. 

We can relate the solutions &j(s) to the singular values of the Hankel operator H. Let vj(s) and 
o;(s) denote the singular vector and the singular-value parameterized by s corresponding to &;(s). Then 
we have 


vi(s) := CT(E;(s)), 
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ae) 


FIGURE 4.2 Configuration of &;(s) and &2(s) in the case n = 2. 


oe IAvi())lltr _— WOKS) Ihas _ [Lo(Ei(s)) 
a vi(s)Ihtp Ct (Ei(s)) Iz. L¢(&;(s))" 


This expression yields an explicit expression of the singular values 0;(s)’s of the Hankel operator H. These 
functions o;(s)’s are called Hankel singular values. Without loss of generality we assume that the following 
equation holds for i= 1,2,...,n in a neighborhood of the origin 


min{o;(s), oj(—s)} > max{oj+1(s), oj41(—s)}. (4.39) 


As in the linear case, the solution curves &;(s) play the role of the coordinate axes of a balanced 
realization. By applying an isometric coordinate transformation which maps the solution curves &;(s)’s 
into the coordinate axes, we obtain a realization whose (new) coordinate axes x; are the solution of 
Equation 4.37, that is, 


OL (x) . OL (x) 


(4.40) 


OX |x=(,...,0,x),0,...0) OX | =(0,...,0,%;,0,---0), 


o;(x;) L,(0,. : . 0, xj, 0,. : .,0) (4.41) 
mM (0,...,05.Xi,0,...5,0)' ; 


Equation 4.41 implies that the new coordinate axes x;, i= 1,...,n are the solutions of Equation 4.37 for 
Hankel singular-value analysis. Therefore, the Hankel norm can be obtained by 


| lla = sup HOO tg = sup max o;(s) 
uzo (UlUllz, <R' 
Lo(x1,0,...,0) 
= sup 


R L-(x1,0,...,0) 


x1E 


provided the ordering condition (Equation 4.39) holds for all s ¢ IR. Furthermore, apply this coordinate 
transformation recursively to all lower dimensional subspaces such as (x1, x2,...,X,,0,...,0), then we 
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obtain a state-space realization satisfying Equation 4.41 and 

OL, (x) ag ves OL¢(x) 
Ox; Ox; 

This property is crucial for balanced realization and model order reduction. Using tools from differential 


topology, for example, [31], we can prove that the obtained realization is diffeomorphic to the following 
precise input-normal/output-diagonal realization. 


x=0 = =0. (4.42) 


Theorem 4.19: [11,14] 


Consider the system X in Equation 4.30. Suppose that the assumptions in Theorem 4.18 hold. Then there 
exists coordinates in a neighborhood of the origin such that the system is in input-normal/output-diagonal 


form satisfying 

] n 

2 

L, (x) = 2 = Xj> 
i=l 
1 n 

Lolx) = 5 2% oi(xj)”. 

i= 


This realization is more precise than the realization in Theorem 4.16 in the following sense: (a) The 
solutions of Equation 4.37 coincide with the coordinate axes, that is, Equation 4.40 holds. (b) The ratio 
of the observability function L, to the controllability function L, equals the singular values o;(x;)’s on 
the coordinate axes, that is, Equation 4.41 holds. (c) An exact balanced realization can be obtained by a 
coordinate transformation 


Zi = di (xj) = xj V o;(%)), (4,43) 


which is well-defined in a neighborhood of the origin. 


Corollary 4.1: [11,14] 


The coordinate change (Equation 4.43) transforms the input-normal realization in Theorem 4.19 into the 
following balanced form: 


pL. 2? 


L@)= 5) 


 o;(z)’ 


] n 
Lola) = 5) zi oi(zi). 
i=1 


Since we only use the coordinate transformation (Equation 4.43) preserving the coordinate axes, the 
realization obtained here also satisfies the properties (a) and (b) explained above. The controllability and 
observability functions can be written as 


1 = 
Le(2) = 52" diag(or (21), 1 Gn(2n)) gz, 
ee 
P(z) 
1 
Lo(z) = 5 z! diag(o1(z1),--.,0n(Zn)) Z- 
eee 


Qt) 
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Here P(z) and Q(z) can be regarded as nonlinear counterparts of the balanced controllability and observ- 
ability Gramians, since 


P(z) = Q(z) = diag(o1 (z1), 62(z2), -.., On(Zn)). 
The axes of this realization are uniquely determined. We call this state-space realization a balanced 


realization of the nonlinear system & in Equation 4.30. As in the linear case, both the relationship between 
the input-to-state and state-to-output behavior and that among the coordinate axes are balanced. 


4.3.3 Model Order Reduction 


An important application of balanced realizations is that it is a tool for model order reduction called 
balanced truncation. Here, a model order reduction method preserving the Hankel norm of the original 
system is proposed. Suppose that the system of Equation 4.30 is balanced in the sense that it satisfies 
Equations 4.41 and 4.42. Note that the realizations in Theorem 4.19 and Corollary 4.1 satisfy these 
conditions. Suppose, that 


min{oz(s), o4(—s)} >> max{oz+1(s), on41(-s)} 


holds with a certain k (1 < k < n). Divide the state into two vectors x = (x%, x”) 


k 
x* :=(x1,...,x,) ER’, 


n—k 
x? = (Xe415---5%Xn) € RR", 


and the vector field into two vector fields accordingly 


and truncate the state by substituting x’ = 0. Then we obtain a k-dimensional state-space model £4 with 
the state x” (with a (n — k)-dimensional residual model D? with the state x°). 


5a. [HFS 0) + 92%, Ou" (4.44) 
"| y4 = h(x4, 0) | 
xb = £60, x6) + g(0,.x')ub 
<i, i me (4.45) 


This procedure is called balanced truncation. The obtained reduced order models have preserved the 
following properties. 


Theorem 4.20: [13,14] 


Suppose that the system & satisfies Equations 4.41 and 4.42 and apply the balanced truncation procedure 
explained above. Then the controllability and observability functions of the reduced order models X° and 
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&° denoted by L2, L?, L%, and L!, respectively, satisfy the following equations: 


L9(x*) =L,(x4,0), L4(x%) = Lo(x*, 0), 
L(x’) =1,(0,x"), L(x’) = L,(0, x°), 


which implies 


of (xi) = oj(x#), i=1,2,...,k, 


0) (x?) = oj44(x?), i=1,2,...,n—k, 


with the singular values o°’s of the system &° and the singular values o° of the system X°. In particular, if 
0 is defined globally, then 
2" = || Zlla- 


Theorem 4.20 states that the important characteristics of the original system such as represented by 
the controllability and observability functions and Hankel singular values are preserved. Moreover, by 
Theorem 4.15, this implies that the controllability, observability, minimality, and the gain property is 
preserved under the model reduction. These preservation properties hold for truncation of any realiza- 
tion satisfying the conditions (Equations 4.41 and 4.42), such as the realizations in Theorem 4.19 and 
Corollary 4.1 [13,14]. Furthermore, concerning the stability, (global) Lyapunov stability and local asymp- 
totic stability are preserved with this procedure as well. Note that this theorem is a natural nonlinear 
counterpart of the linear theory. However, a nonlinear counterpart of the error bound of the reduced 
order model has not been found yet. 


4.3.4 Other Types of Balancing for Nonlinear Systems 


As for linear systems, there exist extensions of LQG, and coprime balancing [49], Hoo or L2-gain balancing 
[46], and positive/bounded real and dissipativity-based balancing [23]. In fact, in [23] a direct relation is 
obtained with Hankel operator analysis for augmented systems. 

The presented work treats balanced realizations for nonlinear systems based on balancing in a (possibly 
large) region around an equilibrium point, where a relation with the Hankel operator, observability and 
controllability operators and functions, and minimality of the nonlinear system is obtained. A drawback 
of these methods is the computational effort that is needed to compute the balanced realization. As 
mentioned in the introduction, other extensions of the linear notion of balancing can be found in for 
example, [20,28,58,59]. 


4.4 Concluding Remarks 


In this Chapter, balanced realizations for linear and nonlinear systems and model reduction based on 
these realizations are treated. There exists a vast amount of literature on this topic, and the reference list 
in this paper is certainly not complete. For example, some of the basics for linear balanced realizations 
can be found in [40], and we did not pay any attention to the balancing methods treating uncertain, and 
time- and parameter-varying linear systems; for example, [4,44], behavioral balancing, for example, [53], 
or the numerical side of balancing, for example, [5]. 

Recently, a lot of interest is taken in structure preserving order techniques for both linear and nonlinear 
systems, where the additional structure to be preserved is a physical structure, such as port-Hamiltonian 
structure, [56], and other physical structures, as mentioned in our linear systems section. For example, 
for circuit simulators with continuously growing orders of the models, a need for interpreting a reduced 
order model as a circuit is important, such as, [50]. Due to the explicit interconnection (input/output 


4-22, Control System Advanced Methods 


like) structure of the circuits, order reduction methods based on balancing are attractive to apply to these 
circuits. However, structure preservation and circuit interpretation of the corresponding reduced order 
models is not possible yet, and is one of the motivators for further research to both linear and nonlinear 
structure preserving order reduction methods. 


References 


1. A.C. Antoulas, Approximation of Large-Scale Dynamical Systems, SLAM, Philadelphia, 2005. 
A. Astolfi, Model reduction by moment matching, Proc. 7th IFAC NOLCOS, Pretoria, South Africa, 
95-102, August 2007. 
3. J. Batt, Nonlinear compact mappings and their adjoints, Math. Ann., 189, 5-25, 1970. 
4, C.L. Beck, J. Doyle, and K. Glover, Model reduction of multi-dimensional and uncertain systems, [EEE 
Trans. Automat. Control, AC-41, 10, 1466-1477, 1996. 
5. P. Benner, V. Mehrmann, and D.C. Sorensen, Eds., Dimension Reduction of i Large-Scale Systems, 
Springer-Verlag, Berlin, 2005. 
6. R. Courant and D. Hilbert, Methods of Mathematical Physics, Vol. 1, Interscience Publishers, New York, 
1953. 
7. U.B. Desaiand D. Pal, A transformation approach to stochastic model reduction, IEEE Trans Automat 
Control, 29, 1097-1100, 1984. 
8. K.V. Fernando and H. Nicholson, Singular pertubational model reduction of balanced systems, [EEE 
Trans. Automat. Control, AC-27, 466-468, 1982. 
9. K. Fujimoto, What are singular values of nonlinear operators?, Proc. 43rd IEEE Conf. on Decision and 
Control, The Bahamas, 1623-1628, 2004. 
10. K. Fujimoto, J. M. A. Scherpen, and W. S. Gray, Hamiltonian realizations of nonlinear adjoint operators, 
Automatica, 38, 10, 1769-1775, 2002. 
11. K. Fujimoto and J.M.A. Scherpen, Nonlinear balanced realization based on singular value analysis of 
Hankel operators, Proc. 42nd IEEE Conf. on Decision and Control, Maui, HI, 6072-6077, 2003. 
12. K. Fujimoto and J.M.A. Scherpen, Nonlinear input-normal realizations based on the differential eigen- 
structure of Hankel operators, IEEE Trans. Automat. Control, AC-50, 1, 2-18, 2005. 
13. K. Fujimoto and J.M.A. Scherpen, Singular value analysis and balanced realizations for nonlinear sys- 
tems, in: Model Order Reduction: Theory, Research Aspects and Applications, Eds. W. Schilders, H. van 
der Vorst, and J. Rommes, Springer-Verlag, Berlin, 251-272, 2008. 
14. K. Fujimoto and J.M.A. Scherpen, Model reduction for nonlinear systems based on the balanced real- 
ization, SIAM J. Control Optim., 48 (7), 4591-4623, 2010. 
15. F.R. Gantmacher, The Theory of Matrices, Chelsea, New York, 1960. 
16. W. Gawronski, Balanced Control of Flexible Structures, Lecture Notes in Contr. Inf. Sc. 211, Springer- 
Verlag, Berlin, 1996. 
17. K. Glover, All optimal Hankel-norm approximations of linear multivariable systems and their L© -error 
bounds, Int. [. Control, 39, 1115-1193, 1984. 
18. W.S. Gray and J.M.A. Scherpen, State dependent matrices in quadratic forms, Systems Control Lett., 44, 
3, 219-232, 2001. 
19. S. Gugercin and A. C. Antoulas, A survey of model reduction by balanced truncation and some new 
results, Int. J. Control, 77 (8), 748-766, 2004. 
20. J. Hahn and T.F. Edgar, An improved method for nonlinear model reduction using balancing of empir- 
ical Gramians, Comp Chem _Eng., 26, 10, 1379-1397, 2002. 
21. P. Heuberger, A family of reduced order models, based on open-loop balancing, Ident. Model. Contr., 
Delft University Press, 1, 1-10, 1990. 
22. J.W. Hoffmann, Normalized coprime factorizations in continuous and discrete time—a joint state-space 
approach, IMA J. Math. Contr. Info. 13, 359-384, 1996. 
23. T.C. Ionescu, K. Fujimoto and J.M.A. Scherpen, Dissipativity preserving balancing for nonlinear 
systems—a Hankel operator approach, Systems and Control Letters, 59, 180-194, 2010. 
24. <A. Isidori, Nonlinear Control Systems (3rd edition), Springer-Verlag, Berlin, 1995. 
25. E.A. Jonckheere and L.M. Silverman, Singular value analysis of deformable systems, J. Circ., Syst, Sign, 
Proc., 1, 447-470, 1982. 
26. E.A.Jonckheere and L.M. Silverman, A new set of invariants for linear systems—Applications to reduced 
order compensator design, IEEE Trans. Automat. Control, AC-28, 953-964, 1983. 
27. T. Kailath, Linear Systems, Prentice-Hall, Englewood Cliffs, NJ, 1980. 


Balanced Realizations 4-23 


28. 


29, 


30. 


31. 
32. 


33. 


34, 


35. 


36. 


37. 


38. 


39. 


40. 


4l. 


42. 


43, 


44. 


45. 
46. 


47. 


48. 


49, 


50. 


51. 
52. 


53; 


54. 


55. 


56. 


57. 


58. 


S. Lall, J.E. Marsden, and S. Glavaski, A subspace approach to balanced truncation for model reduction 
of nonlinear control systems, Int_] Robust Nonlinear Control, 12, 6, 519-535, 2002. 

D.G. Meyer, Fractional balanced reduction—model reduction via a fractional representation, [EEE 
Trans. Automat. Control, AC-26, 1341-1345, 1990. 

D.G. Meyer and S. Srinivasan, Balancing and model reduction for second order form linear systems, 
IEEE Trans. Automat. Control, AC-41, 11, 1632-1644, 1996. 

J. W. Milnor, Topology from the Differential Viewpoint, Princeton University Press, Princeton, NJ, 1965. 
B.C. Moore, Principal component analysis in linear systems: Controllability, observability and model 
reduction, IEEE Trans, Automat. Control, AC-26, 17-32, 1981. 

D. Mustafa, Hoo-characteristic values, Proc. 28th IEEE Conf. on Decision and Control, Tampa, FL, 
1483-1487, 1989. 

D. Mustafa and K. Glover, Minimum Entropy Hoo Control, Lect. Notes Contr. Inf. Sci. No. 146, Springer- 
Verlag, Berlin, 1990. 

D. Mustafa and K. Glover, Controller reduction by Hoo-balanced truncation, IEEE Trans. Automat. 
Control, AC-36, 668-682, 1991. 

C.N. Nett, C.A. Jacobson, and M.J. Balas, A connection between state-space and doubly coprime frac- 
tional representations, IEEE Trans. Autom. Control, AC-29, 831-832, 1984. 

A.J. Newman and P.S. Krishnaprasad, Computation for nonlinear balancing, Proc. 37th IEEE Conf. on 
Decision and Control, Tampa, FL, 4103-4104, 1998. 

A. Newman and P. S. Krishnaprasad, Computing balanced realizations for nonlinear systems, Proc. 
Symp. Mathematical Theory of Networks and Systems, 2000. 

R. Ober and D. McFarlane, Balanced canonical forms for minimal systems: A normalized coprime factor 
approach, Linear Algebra and its Applications, 122-124, 23-64, 1989. 

G. Obinata and B.D.O. Anderson, Model Reduction for Control System Design, Springer-Verlag, London, 
2001. 

P.C. Opdenacker and E.A. Jonckheere, LQG balancing and reduced LQG compensation of symmetric 
passive systems, Int J Control, 41, 73-109, 1985. 

P. C. Opdenacker and E. A. Jonckheere, A contraction mapping preserving balanced reduction scheme 
and its infinity norm error bounds, IEEE Trans_Circuits Systems 35 (2), 184-189, 1988. 

L. Pernebo and L.M. Silverman, Model reduction via balanced state space representations, IEEE Trans 
Automat. Control, AC-27, 382-387, 1982. 

H. Sandberg and A. Rantzer, Balanced truncation of linear time-varying systems, JEEE Trans. Automat, 
Control, 49, 2, 217—229, 2004. 

J.M.A. Scherpen, Balancing for nonlinear systems, Systems Control Lett., 21, 143-153, 1993. 

J.M.A. Scherpen, Ho balancing for nonlinear systems, Int. J. Robust Nonlinear Control, 6, 645-668, 
1996. 

J.M.A. Scherpen and W.S. Gray, Minimality and local state decompositions of a nonlinear state space 
realization using energy functions, IEEE Trans. Automat. Control, AC-45, 11, 2079-2086, 2000. 

J.M.A. Scherpen and W.S. Gray, Nonlinear Hilbert adjoints: Properties and applications to Hankel 
singular value analysis, Nonlinear Anal., Theory, Methods Appl., 51, 5, 883-901, 2002. 

J.M.A. Scherpen and A.J. van der Schaft, Normalized coprime factorizations and balancing for unstable 
nonlinear systems, Int. J. Control, 60, 6, 1193-1222, 1994. 

W.H.A. Schilder, H.A. van der Vorst, and J. Rommes, Eds., Model Order Reduction: Theory, Research 
Aspects and Applications, Mathematics in Industry, Vol. 13, Springer, Berlin, 2008. 

G.W. Stewart, On the early history of the singular value decomposition, SIAM Rev., 35, 4, 551-566, 1993. 
H.L. Trentelman, Bounded real and positivey real balanced truncation using & normalized coprime 
factor, System Control Lett., 58, 12, 871-879, 2009. 

H.L. Trentelman and P. Rapisarda, A behavioral approach to passivity and bounded realness preserving 
balanced truncation with error bounds, Proc. 47th IEEE Conf. on Decision and Control, Shanghai, China, 
2009. 

A.J. van der Schaft, Balancing of lossless and passive systems, IEEE Trans. Automat. Control, AC-53, 
2153-2157, 2008. 

A.J. van der Schaft and J.E. Oeloff, Model reduction of linear conservative mechanical systems, IEEE 
Trans Automat Control AC-35, 729-733, 1990. 

A.J. van der Schaft and R. Polyuga, Structure-preserving model reduction of complex physical systems, 
Proc. 48th IEEE Conf. on Decision and Control, Shanghai, China, December 16-18, 2009. 

E. Verriest, Low sensitivity design and optimal order reduction for the LQG-problem, Proc. 24th Midwest 
Symp. Circ. Syst. Albuquerque, NM, 1981, 365-369. 

EL. Verriest and W.S. Gray, Flow balancing nonlinear systems, Proc. 2000 Int. Symp. Math. Th. Netw. 
Syst. (MTNS), 2000. 


4-24 


59. 


60. 
6l. 


62. 


63. 


64. 


Control System Advanced Methods 


E.I. Verriest and W.S. Gray, Balanced nonlinear realizations, Proc. 43rd IEEE Conf. on Decision and 
Control, The Bahamas, 1164-1169, 2004. 

M. Vidyasagar, Control System Synthesis: A Factorization Approach, MIT Press, Cambridge, 1985. 

S. Weiland, Theory of Approximation and Disturbance Attenuation for Linear Systems, Doctoral disser- 
tation, University of Groningen, 1991. 

P. Wortelboer, Frequency- Weighted Balanced Reduction of Closed Loop Mechanical Servo Systems: The- 
ory and Tools, doctoral dissertation, Delft University of Technology, 1994. 

K. Zhou, J. C. Doyle, and K. Glover, Robust and Optimal Control, Prentice-Hall, Inc., Upper Saddle 
River, NJ, 1996. 

A. J. Krener, Reduced order modeling of nonlinear control systems, in Analysis and Design of Nonlinear 
Control Systems, A. Astolfi and L. Marconi, eds., Springer-Verlag, 41-62, 2008. 


Geometric Theory of 
Linear Systems* 


MYM INcases esis cs css ah css das tasers seatercueniuicencs 

5.2 Review of Elementary Notions 

5.3. (A, imB)-Controlled and (A, ker C)- 
Conditioned Invariant Subspaces and 


DIMOY wacaiscci aca enn nceienamnananaiontens 5-6 
5.4 Algebraic Properties of Controlled and 

Conditioned Invariants .........cccccsseseseeeseeeeeees 5-10 
5.5 Maximum-Controlled and Minimum- 

Conditioned TAVar esis scsssecersaccaacasesnesacencnsetee 5-11 


5.6  Self-Bounded-Controlled and Self-Hidden- 
Conditioned Invariants and Constrained 
Reachability and Observability «0.0... 

5.7 Internal, External Stabilizability 

5.8 Disturbance Localization (or Decoupling) 


PPG DREN ods sasscivasnsoicnataees paeanncnaranainainemieanies 5-17 
5.9 Disturbance Localization with Stability .......... 5-19 
5.10 Disturbance Localization by Dynamic 
CRUG ISAT Ss iaiciorhleeien cee aBnaiataiansineen 5-20 
Fumio Hamano Sob CGR SIE siesta icuahabencisaecdatasieandencenas 5-25 
California State University RETA Sto 25:0s eta icaies eoieniadneaioucles beeeneaebnuodels 5-25 


5.1 Introduction 


In the late 1960s, Basile and Marro (1969) (and later Wonham and Morse, 1970) discovered that the 
behavior of time-invariant linear control systems can be seen as a manifestation of the subspaces similar 
to the invariant subspaces characterized by the system matrices. As a result, the system behavior can 
be predicted and the solvability of many control problems can be tested by examining the properties of 
such subspaces. In many instances, one can understand essential issues intuitively in geometric terms. 
Moreover, thanks to good algorithms and software available in the literature (see Basile and Marro, 
1992), the above subspaces can be generated and the properties can be readily examined by using per- 
sonal computers. Thus, a large class of problems involving feedback control laws and observability of 
linear systems can be solved effectively by this geometric method, for example, problems of disturbance 
localization, decoupling, unknown input observability and system inversion, observer design, regulation 


* This chapter is dedicated to Professors G. Basile and G. Marro for their pioneering contributions in the development of 


geometric methods for linear systems. 
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and tracking, model following, robust control, and so on. Comprehensive treatments of the basic theory 
and many applications, including the ones mentioned above, can be found in the excellent books by 
Wonham (1985), Basile and Marro (1992), and Trentelman et al. (2002), with additional later results 
found in the newer books, for example, controlled and conditioned invariant subspaces and duality 
are treated in an organized fashion in the 2nd book and, in addition to the duality, an extension of 
the theory to include distributions as inputs and the relation between disturbance localization and H2 
optimal control problem are also given in the 3rd book. The method is also useful in the analysis and 
design of decentralized control systems (Hamano and Furuta, 1975), perfect and near-perfect signal 
decoupling (and its applications to other control problems) for nonminimum-phase systems (Marro 
and Zattoni, 2006), and failure/fault detection (Massoumnia et al., 1989). This chapter serves as an 
introduction to the subject. Regrettably, citations in this chapter are rather limited due to editorial 
restrictions. Extensive references can be found in the above-mentioned books as well as in the chapter 
by Marro (2008). To prepare the present chapter, the book by Basile and Marro (1992) has been used 
as the primary reference, and the majority of the proofs omitted in this chapter can be found in this 
reference. 

Section 5.2 gives a review of elementary notions including invariant subspaces, reachability, control- 
lability, observability, and detectability. It also provides convenient formulae for subspace calculations. 
Sections 5.3 through 5.7 describe the basic ingredients of the geometric theory (or approach). More 
specifically, Section 5.3 introduces the fundamental notions of (A, im B)-controlled and (A, ker C)- 
conditioned invariants (which are subspaces of the state space), and Section 5.4 provides some algebraic 
properties of these invariants. In Section 5.5, “largest” controlled and “smallest” conditioned invariants 
are presented with respect to certain subspaces and Section 5.6 discusses well-structured special classes of 
controlled and conditioned invariants. Section 5.7 analyzes the above invariants in relation to stabiliza- 
tion. Sections 5.8 through 5.10 describe applications to demonstrate the use of the basic tools developed 
in the previous sections. For this goal, the disturbance localization problem is chosen and it is discussed 
in three different situations with varying degrees of sophistication. The disturbance localization prob- 
lems are chosen since the methods used to solve the problems can be used or extended to solve other 
more involved problems. It also has historical significance as one of the first problems for which the 
geometric method was used. Section 5.11 provides the conclusion and brief statements about practi- 
cal applications and about extensions of geometric notions and approach to more general or different 
systems. 


Notation: Capital letters A, B, and so on denote the matrices (or linear maps) with I and I, reserved, 
respectively, for an identity matrix (of appropriate dimension) and an n x n identity matrix. The transpose 
of a matrix A is denoted by A’. Capital script letters such as V, W represent vector spaces or subspaces. 
Small letters x, y, and so on are column vectors. Scalars are also denoted by small letters. The letter 0 is 


ee) 


used for a zero matrix, vector, or scalar depending on the context. Notation “:=” means “(the left-hand 
side, i.e., ‘: side) is defined by (the right-hand side, i.e., ‘=’ side).” Similarly for “=:” where the roles of 
the left- and right-hand sides are reversed. The image (or range) and the kernel (or null space) of M are 
denoted by im M and ker M, respectively. The expression V + W represents the sum of two subspaces 
Y and W, that is, V+W:={v+w:veVandweW}. If V is a subspace of W, we write VC W. 
If VC &, we use AV := {xe X: Ax € V}, that is, the set of all x ¢ ¥ satisfying Ax € V. Similarly, 


A*y:={xeX: Akx eV} ,k=1,2,.... 


5.2 Review of Elementary Notions 


In this section we will review invariant subspaces and some of their basic roles in the context of the linear 
systems. 
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Definition 5.1: 


A subspace V of X := R" is said to be A-invariant if 
AY CY, (5.1) 


that is,x EV > Ax EY. 


An A-invariant subspace plays the following obvious but important role for an autonomous linear 
system. Consider the autonomous linear system described by 


x(t) = Ax(t), x(0) = Xo. (5.2) 


where the column vectors x(t) € V := R” and x, € ¥ are, respectively, the state of the system at time t > 0 
and the initial state, and A is an x n real matrix. Now, suppose a subspace V is A-invariant. Clearly, 
if x(t) € V, then the rate of change x(t) € V, which implies that the state remains in V. More strongly, 
we have 


Lemma 5.1: 


Let V C X. For the autonomous linear system described by Equation 5.2, Xx» € V implies x(t) € V for all 
t > 0 ifand only if V is A-invariant. 


Let V be A-invariant, and introduce a new basis {e1,..., en}, such that 
span{e},...,ey}=V, v<n. (5.3) 


Define a coordinate transformation by 


x= [ey oe en] Bi se R", XY € RY, X2 ERY, (5.4) 


Then, it is easy to see that, with respect to the new basis, the state Equation 5.2 can be rewritten as 


xi(t)]  fAn Ar] [xa(2) 
Bal 7 0 32 Ba : 2) 
X1(0) = X10, X2(0) = X29. (5.6) 


Clearly, if x29 = 0, then x(t) = 0 for all t > 0, that is, x» € V implies x(t) € V for all t > 0 (which has 
been stated in Lemma 5.1). 

Let V be an A-invariant subspace in 4’. The restriction A| V of a linear map A: ¥ > ¥ (orannxn 
real matrix A) to a subspace V is a linear map from V to V, mapping vt Av for all ve V. For x € 4, 
we write x + V:={x+v:ve VY} called the coset of x modulo Y, which represents a hyperplane passing 
through a point x. The set of cosets modulo V is a vector space called the factor space (or quotient space) 
and it is denoted by 4’/V. An induced map A|X’/V isa linear map defined byx + Vbh> Ax+V, x€ X. 

An A-invariant subspace V is said to be internally stable if Ay, in Equation 5.5 is stable, that is, all the 
eigenvalues have negative real parts, or equivalently, if A| V is stable. Therefore, x(t) converges to the 
zero state as tf > oo whenever x, € V if and only if V is internally stable. Also, an A-invariant subspace V 
is said to be externally stable if Ap, is stable, that is, if A| /¥V is stable. Clearly, x2(t) converges to zero as 
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t — oo, that is, x(t) converges to V as t > oo ifand only if V is externally stable. Note that the eigenvalues 
of Aj,and Az. do not depend on a particular choice of coordinates (as long as Equation 5.3 is satisfied). 
Let us now consider a continuous time, time-invariant linear control system & := [A,B,C] described 


by 
x(t) = Ax(t)+ Bu(t), x(0) = x9, (5.7) 
y(t) = Cx(t), (5.8) 


where the column vectors x(t) € VY := R", u(t) € R™, and ¢(t) € RP are, respectively, the state, input, 
and output of the system at time t > 0,x, € R” is the initial state, and A,B, and C are real matrices 
with consistent dimensions. We assume that u(t) is piecewise continuous. We are also interested in the 
closed-loop system, namely, we apply a linear state-feedback law of the form 


u(t) = Fx(t), (5.9) 


then Equation 5.7 becomes 
X(t) = (A+ BF)x(t), x(0) =X», (5.10) 


where F is an m x n real matrix. 

Some invariant subspaces are associated with reachability (controllability) and observability. A state x 
is said to be reachable (or controllable) if there is a control which drives the zero state to x (or, respectively, 
x to the zero state) in finite time, that is, if there is u(t),0 <t < tf such that x(0) = 0 (or, respectively, 
x) and x(tf) =X (or, respectively 0) for some 0 < tf <0 (see Kalman et al., 1969, Chapter 2). The set 
of reachable (or controllable) states forms a subspace and it is called the reachable (or, respectively, 
controllable) subspace, which will be denoted as V;eqcy (or, respectively, Veontr). For an n x n matrix M 
and a subspace Z C 4, define 


R(M,Z):=I+MI+4+---+M""'T. 


The reachable and controllable subspaces are characterized by the following. 


Theorem 5.1: 


For the continuous-time system & := [A, B, C], 


Vreach = RIA, im B) =im [B AB... A” 1B] F 


for any m x n real matrix F. Furthermore, 
Vreach = Veontr: (5.12) 


Remark 


The subspace Vyeach = R(A, im B) is the minimum A-invariant subspace containing im B. It is also (A + 
BF)-invariant for any m x n real matrix F. (For a discrete-time system, Equation 5.12 does not hold; 
instead we have Vyeach C Veontr-) 

The pair (A,B) or system ¥ := [A, B, C] is said to be reachable (or controllable) if Vreach = ¥ (or, 
respectively, Veontr = V). The set A := {Xy,...,n} of complex numbers is called a symmetric set if, 
whenever ; is not a real number, 4; = dF for some j = 1,..., where 7 is the complex conjugate of ij. 
Denote by o(A + BF) the spectrum (or the eigenvalues) of A + BF. Then, we have 
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Theorem 5.2: 


For any symmetric set A := {h1,...,n} of complex numbers d1,...,n there isan m x n real matrix F, 
such that o(A + BF) = A if and only if the pair (A, B) is reachable (or controllable). 


Proof. See Wonham (1985); Basile and Marro (1992); and Heymann (1968). 


Remark 


Let dim Vyeach = 7 <n. For any symmetric set A; := {h1,...,,} of complex numbers ij,..., ,, there 
is an mx n real matrix F, such that o(A+ BF| Vyeqcn) = Ar. In fact, since Vege, is A-invariant, using 
the same state coordinate transformation given by Equations 5.3 and 5.4 with V = Vj.q-, and v = r, we 


obtain, from Equation 5.7, 
x1 (t) = An Ai] [x(t) By 
fo)=[o) ae] Exe] *Lo] 


It is easy to show dim im [é Ay, By... Ary"B1| =r, that is, the pair (Aj1, By) of the r-dimensional 
subsystem is reachable. The eigenvalue assignment for V,eq-h follows by applying Theorem 5.2. 

The pair (A, B) is said to be stabilizable if there is a real matrix F, such that the eigenvalues of A+ BF 
have negative real parts. We have (see Basile and Marro, 1992). 


Corollary 5.1: 


Pair (A, B) is stabilizable if and only if Vyeach is externally stable. 


The state x of the system © is said to be unobservable if it produces zero output when the input is not 
applied, that is, ifx(0) = x and u(t) = 0 for all t > 0 implies y(t) = 0 for all t > 0. The set of unobservable 
states forms a subspace which is called the unobservable subspace. This will be denoted by Synops. 


Theorem 5.3: 
C C 
CA C(A+ GC) 
Sunobs = ker é = ker m 
cA} C(A+ Gc)! 
=—ker CNA ker CN---N AT" ker C 
= ker CN (A+ GC)! ker CN---A(A+ GC)" ker C (5.13) 


for any n x p real matrix G. 


Remark 


The subspace Syyop5 is A-invariant. It is also (A + GC)-invariant for any n x p real matrix G. 

The pair (A, C) or system © is said to be observable if Synops = {0}, and the pair (A, C) is said to be 
detectable if A+ GC is stable for some real matrix G. For observability and detectability we have the 
following facts. 
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Theorem 5.4: 


For any symmetric set A := {X1,..-; dn} of complex numbers \1,..., dn; there is an n x p real matrix G 
such that o(A + GC) = A if and only if the pair (A, C) is observable. 


Corollary 5.2: 
The pair (A, C) is detectable if and only if Synops is internally stable. 


The following formulae are useful for subspace calculations. 


Lemma 5.2: 
Let V, V1, V2, V3 C X. Then, letting ylea {x EX: x'v=O0forallve V}, 


Vo =v, 
(Y.+¥.)t =vinvyz, 
(UNY,)t =Vvi+vVz, 
A(V1 + V2) = AV, + AV2, 
A(VNV2) C AV, NAV 2, 


(Ay + A2)V = A1V + A2V, where A; and Az aren xX n matrices, 
(ay)t =a’ yt, 
Vi + (V2 V3) C (V1 +2) (V1 + V3); 


ViN (V2 + V3) D V1 A V2) + Vi YV3), 


ViN(2 +3) = V2 + (VN V3), provided VY, D> v2. 


5.3 (A, im B)-Controlled and (A, ker C)-Conditioned Invariant 
Subspaces and Duality 


In this section, we introduce important subspaces associated with system L :=[A, B, C] described by 
Equation 5.7 (or Equation 5.10) and Equation 5.8 with a state feedback law (Equation 5.9). According 
to Lemma 5.1, for an autonomous linear system described by Equation 5.2, an A-invariant subspace is a 
subspace having the property that any state trajectory starting in the subspace remains in it. However, for 
alinear system X := [A, B, C] with input, A-invariance is not necessary in order for a subspace to have the 
above trajectory confinement property. In fact, let V C ¥. A state trajectory can be confined in V if and 
only if x(t) € V, and to produce x(t) € V whenever x(t) € V, we need x(t) = Ax(t) + Bu(t) = v(t) EV, 
that is, Ax(t) = v(t) — Bu(t) for some u(t) and v(t), which implies AV C Y+ im B. The converse also 
holds. Summarizing, we have 
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Lemma 5.3: 


Consider the system described by Equation 5.7. For each initial state x9 € V, there is an (admissible) input 
u(t), t > 0 such that the corresponding x(t) € V for all t > 0 if and only if 


AVCV+imB. (5.14) 


Subspaces satisfying Equation 5.14 play a fundamental role in the geometric approach, and we introduce 


Definition 5.2: 


A subspace Y is said to be an (A, im B)-controlled invariant (subspace) (Basile and Marro, 1969, 1992) or 
an (A, B)-invariant subspace (Wonham and Morse, 1970; Wonham, 1985) if it is A-invariant modulo imB, 
that is, if Equation 5.14 holds. 


An important property of the above subspace is described by 


Theorem 5.5: 


Let V C X. Then, there exists an m x n real matrix F, such that 
(A+ BF)V CY, (5.15) 
if and only if V is an (A, im B)-controlled invariant. 


Remark 


If the state feedback control law (Equation 5.9) is applied to the system & := [A, B, C], the corresponding 
state equation is described by Equation 5.10. Therefore, recalling Lemma 5.1, if V is an (A, im B)-controlled 
invariant, then there is an F for Equation 5.10 such that x(t) € V for all t > 0, provided x, € V. 

Another class of important subspaces is now introduced (Basile and Marro 1969, 1992). 


Definition 5.3: 


A subspace S of X is said to be an (A, ker C)-conditioned invariant (subspace), if 


A(SMkerC) cS. (5.16) 


There is a duality between controlled invariants and conditioned invariants in the following sense. By 
taking the orthogonal complements of the quantities on both sides of Equation 5.16, we see that Equation 
5.16 is equivalent to {A (SN ker C)}+ D S+, which, in turn, is equivalent to A’! (S'+imC’) 5 St, 
that is, A’St C St +imC’. Similarly, Equation 5.14 holds if and only if A’(V+ Nker B’) c V+. Thus, 
we have 
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Lemma 5.4: 


A subspace S is an (A, ker C)-conditioned invariant if and only if St is an (A’, im C’)-controlled invariant. 
Also, a subspace V is an (A, im B)-controlled invariant if and only if V+ is an (A', ker B’)-conditioned 
invariant. 


Due to the above lemma, the previous theorem can be translated easily into the following property 
(Basile and Marro, 1969, 1992). 


Theorem 5.6: 


Let S C &. Then, there exists an n x p real matrix G satisfying 
(A+ GC)S CS, (5.17) 


if and only if S is an (A, ker C)-conditioned invariant. 


The conditioned invariants are naturally associated with dynamic observers (not necessarily asymp- 
totic). The following Lemma 5.5 summarizes the meaning of the invariants in this context. 

Consider the system & := [A, B, C] described by Equations 5.7 and 5.8. To focus on the state observation 
aspect of the system, we assume that u(t) = 0 for all ¢ > 0. (It is straightforward to extend the result to 
include the nonzero measurable input.) We define the full-order and reduced order observers ©); and 
Lobs,red> Lespectively, as follows: 


Loos : X(t) = (A + GC) X(t) — Gy(t) = (A+ GC) X(t) — GCx(¢), 
Lobs,red | Z(t) = Nz(t) — My(t) = Nz(t) — MCx(t). 


Here, x(t) and z(t) are column vectors with dimensions n and nz (to be determined), respectively, and 
N and M are, respectively, nz x nz and nz x p real matrices. Then, we have 


Lemma 5.5: 


Let S C X := R". The following statements are equivalent: 
i. Sis (A, ker C)-conditioned invariant, that is, Equation 5.16 holds. 
ii. There exists a real matrix G with an appropriate dimension and an observer Xops such that 
x(0) = x(0)(mod S), that isx(0)— x(0) ES 
implies 
x(t) = x(t)(mod S), that is x(t)—x(t)ES, forallt >0. 


iii. There exist real matrices M, N, and H, satisfying ker H = S, with suitable dimensions and an observer 
Lobs,red such that 
z(0) = Hx(0) > z(t) = Hx(t), forallt >0. 


Remark 


The statement (iii) in the above lemma is used by Willems (1981) as the definition of the conditioned 
invariant. Also, see Trentelman et al. (2001, Section 5.1). 


Geometric Theory of Linear Systems 5-9 


Proof of Lemma 5.5. (i) => (ii) : Suppose Equation 5.16 holds. In view of Theorem 5.6 choose G, so that 
Equation 5.17 holds. Define the observer error by e,;(t) := x(t) — x(t). The equations for © and Yop; 
lead to é,,(t) = (A+ GC)e,;(t). Since S is (A+ GC)-invariant, e,-(0) € S > e,(t) ES for all t>0. 
(ii) = (iii): Basically, a modulo S version of Xp; in (ii) is a reduced order observer Yops,red- TO be 
explicit, choose a new basis {e1, ease pgar ees en} for X := R" so that {e1, buts ere is a basis of subspace 
SC X:=R", where ng := dim S. Define T := [er Ong + -en| and apply the state coordinate trans- 
formation x(t) = Tx(t) to © (with zero inputs) to obtain y: x(t) = T 'ATX(t), y(t) = CTx(t), where 
x(t) = [x1 (t) x(t)] x1 (t) € R"S, and X2(t) € R"~"S. We apply the same coordinate transformation to 


Lops» that is, ¥(t) = T(t), to obtain Ey, : ¥(t) = T~!(A + GC)T%(t) — T~!Gy(t), which has the follow- 


ing structure: 
ee X(t) - ea qo] X(t) s [ei] 0 
< X(t) 0 Ag22} | X2(t) G ; 
where R(t) E€ R"S, %5(t) E€ R™-"S, Agcy e Remns)x(n—ns) G) E€ Ri&mns) xP, and Agu Ac, and Gy 
with consistent dimensions. Note that the 0 block in matrix T~!(A+GC)T is due to (A+GC)— 


invariance of S. The corresponding observer error [é,.(t) é (t)] — [Rw R| _ EAG) x(t) 


| a es fo] | 
err2(t) 0 = Aga} Lerra(t)} 


The reduced order observer ops, req can be chosen as follows: 


satisfies 


Lobsred 1 Xa(t) = Ag22%2(t) — Gry(t). 


Define H by H := [0 heal T—!. Then, 


%o(0) — Hx(0) = X2(0) — X2(0) = @,2(0) = 0 


implies 


Ro(t) — Hx(t) = 34(t) — ¥2(t) = 8 yp2(t) = e402 89(0) = 0. 


(iii) => (i) : Let x(0) = x» € SM ker C and choose z(0) = Hx, = 0. Then, we have z(0) = Nz(0) — My(0) = 
Nz(0) — MCx(0) = 0. Since z(t) = Hx(t) this implies Hx(0) = HAx(0) = HAx, =0, that is, Ax € 
ker H = S (for every %) € SN ker C). QED 


A subspace may be both controlled and conditioned invariant. In such a case, we have the following 
(Basile and Marro, 1992; Hamano and Furuta, 1975): 


Lemma 5.6: 


There exists an m x p real matrix K, such that 
(A+ BKC)Y CY, (5.18) 


if and only if V is both an (A, im B)-controlled invariant and an (A, ker C)-conditioned invariant. 


Proof. [Only if part]. The controlled invariance and conditioned invariance follow trivially by Theorems 
5.5 and 5.6, respectively. 

[If part]. For C= 0 or ker C=0 or V = 0, the statement of the lemma trivially holds. So we prove the 
lemma assuming such trivialities do not occur. Since by assumption Y is an (A, im B)-controlled invariant, 
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there is an m x n real matrix F satisfying (A + BF)V C V. Assuming V MN ker C # {0}, let {v1 es vy} bea 
basis of VM ker C. Complete the basis {v1 fa Vt Wyo ds vn} of ¥ in such a way that {v1 sgt Vees vy} is 
a basis of V where 1<  < v <n. Define an m x n real matrix F by 


Fy, i=pwt+l,...,v 
Fv; = 30, i=1,...,p (5.19) 
arbitrary, otherwise 


Choose K so that F = KC, that is, F [Mi pte vy | =K [Cry fis Cry]. Note that due to the particular 
choice of basis, the columns of [Cyyp4i Sie Cry | are linearly independent, and so the above K certainly 
exists. To show that Equation 5.18 holds, let v € V. Then, v = Sea avi + i 4, 4i¥j for some numbers 
a‘s. Therefore, by the above choice of K, 


IL v 
(A+ BKC)v=A)ajvit(A+BF) > aij. 
i=l i=utl 
But, the first sum belongs to V by Equation 5.16 (where S = V) and the second sum is an element of V by 
the choice of F. Thus, (A+ BKC)v € V. 
If VN ker C = {0}, the basis for VM ker C is empty and Equation 5.19 can be modified by dropping the 
second row and setting jt = 0. Equation 5.18 follows similarly to the above. QED 


5.4 Algebraic Properties of Controlled and Conditioned Invariants 


An (A, im B)-controlled invariant has the following properties in addition to the ones discussed in the 
previous section. The proofs are omitted. They can be found in Basile and Marro (1992, Chapter 4). 


Lemma 5.7: 


If V, and V2 are (A, im B)-controlled invariants, so is Vj + V2. 


Remark 


But, the intersection of two (A, im B)-controlled invariants is not in general an (A, im B)-controlled 
invariant. 


Lemma 5.8: 


Let VY, and V2 be (A, im B)-controlled invariants. Then, there is an m x n real matrix F satisfying 
(A+ BF)V; CV; i=1,2, (5.20) 


if and only if V) N V2 is an (A, im B)-controlled invariant. 


By duality, we have 


Lemma 5.9: 


If S, and S are (A, ker C)-conditioned invariants, so is Sy NS. 
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Remark 


S + S2 is not necessarily an (A, ker C)-conditioned invariant. 


Lemma 5.10: 


Let S, and S» be (A, ker C)-conditioned invariants. Then, there is an n x p real matrix G satisfying 
(A+ GC)S;C §;, i=1,2, (5.21) 


if and only if S| + So is an (A, ker C)-conditioned invariant. 


5.5 Maximum-Controlled and Minimum-Conditioned Invariants 


Let K C #, and consider the set of (A, im B)-controlled invariants contained in K (by subspace inclusion). 
Lemma 5.7 states that the set of (A, im B)-controlled invariants is closed under subspace addition. As 
a result, the set of (A, im B)-controlled invariants contained in K has a largest element or supremum. 
This element is a unique subspace that contains (by subspace inclusion) any other (A, im B)-controlled 
invariants contained in K, and is called the maximum (or supremum) (A, im B)-controlled invariant 
contained in K. This is denoted in the sequel as Vnax(A, im B, XK). Similarly, let Z C ¥Y. Then, owing 
to Lemma 5.9, it can be shown that the set of (A, ker C)-conditioned invariants containing Z has a 
smallest element or infimum. This is a unique (A, ker C)-conditioned invariant contained in all other 
(A, ker C)-conditioned invariants containing Z, and is called the minimum (or infimum) (A, ker C)- 
conditioned invariant containing Z. This subspace will be denoted as Smin(A, ker C, Z). The subspaces 
Vmax(A, im B,K) and Spin(A, ker C, Z) are important because they can be computed in a finite num- 
ber of steps (in at most n iterations) and because testing the solvability of control problems typically 
reduces to checking the conditions involving these subspaces. The geometric algorithms to compute 
Vmax(A, im B, ) and Smin(A, ker C, Z) are given below. 
Algorithm to compute Vmax(A, im B, K): 


Vinax (A, im B, K) = Vadim K> (5.22a) 

where 
Vo =K, (5.22b) 
Vj;=KNA1(Vj-1+imB), i=1,...,dimK. (5.22c) 


Proof of Equation 5.22a. See Basile and Marro (1992) or Wonham (1985). 


Remark 

The algorithm defined by Equations 5.22 has the following properties: 
i Vi DVWD-+-++D Vaimk- 
ii, If Ve = Vey), then Ve = Vey) =--- = Vaimk- 

Remark 


For an algorithm to compute F such that (A + BF)Vmax(A, im B,C) C Vmax(A, im B, K), see Basile and 
Marro (1992). 
The following is the dual of the above algorithm (see Basile and Marro, 1992). 
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Algorithm to calculate Smin(A, ker C, Z): 


Smin(A, ker C, Z) = Sp_dimT> (5.23a) 
where 
So =f, (5.23b) 
S;=I+A(Sj-;NkerC), i=1,...,n—dimZ. (5.23c) 
Remark 


The algorithm generates a monotonically nondecreasing sequence: 


(i) Sy CS2 C++: C Sy-dimt- 
(ii) IfS~g = Se41, then Sp = Sey) = +++ =Sy_dimz- 


5.6 Self-Bounded-Controlled and Self-Hidden-Conditioned 
Invariants and Constrained Reachability and Observability 


Let V, be an (A, im B)-controlled invariant contained in K, and consider all possible state trajectories 
(with different controls) starting at x» in V, and confined in K. We know that there is at least one control 
for which the state trajectory remains in Vo, but that there may be another control for which the state 
trajectory goes out of V, while remaining in K. However, some (A, im B)-controlled invariant contained 
in K, say V, has a stronger property that, for any initial state in V, there is no control which drives the 
state (initially in V) out of V while maintaining the state trajectory in K, that is, the state trajectory must 
go out of K if it ever goes out of V C K. The subspace Vmax (A, im B, K) is one of those since it contains 
all states that can be controlled to remain in K. Such an (A, im B)-controlled invariant V contained in K 
is in general characterized by 

Vinax(A, im B,K)N imBC V, (5.24) 


and we have (Basile and Marro, 1982, 1992) 


Definition 5.4: 


An (A, im B)-controlled invariant V contained in K is said to be self-bounded with respect to K if Equation 
5.24 holds. 


Remark 


The left-hand side of inclusion 5.24 represents the set of all possible influences of control on the state 
velocity at each instant of time that do not pull the state out of K. 

Self-bounded (A, im B)-controlled invariants have the following properties. 

For each K we can choose a single state-feedback control law, which works for all the self-bounded 
(A, im B)-controlled invariants with respect to K. More precisely, 


Lemma 5.11: 


Let F bean m x n real matrix satisfying 


(A+ BF)Vmax(A, im B,K) C Vmax(A, im B, K). (5.25) 
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Then, every self-bounded (A, im B)-controlled invariant V with respect to K satisfies 
(A+ BF)V CY. (5.26) 


Proof. See Basile and Marro (1992). 


It can be shown that the set of self-bounded (A, im B)-controlled invariants in K is closed under 
subspace intersection, that is, if Vy) and V2 are self-bounded (A, im B)-controlled invariants with respect 
to K, so is Vj NV2. Therefore, the above set has a minimum element which is called the minimum self- 
bounded (A, im B)-controlled invariant with respect to K (Basile and Marro, 1982, 1992), denoted in 
this chapter by Vsp,min(A, im B, K). The subspace Vp, min(A, im B, ) is related to Vmax(A, im B, K) and 
Smin(A, K, im B) as follows. 


Theorem 5.7: 


Vsb,min (A, im B, K) = Vmnax(A, im B, K) a) Smin(A, K, im B). (5.27) 


Proof. See Basile and Marro (1992, Chapter 4) and Morse (1973). 

The minimum self-bounded (A, im B)-controlled invariant is closely related to constrained reachability. 
The set of all states that can be reached from the zero state through state trajectories constrained in K in 
finite time is called the reachable set (or reachable subspace, since the set forms a subspace) in K (Basile 
and Marro, 1992). It is also called the supremal (A, B)-controllability subspace contained in K (Wonham, 
1985). This set will be denoted by V;eacn(K). Clearly, it has the following properties. 


Lemma 5.12: 


Vreach(C) C Vmax(A, im B,K) C K, (5.28) 
Vreach Vmax(A, im B, K)) = Vreach(K). (5.29) 


To examine Vea; (AK) further, let F be a matrix satisfying Equation 5.25. It is easy to observe that the set 
of x(t) (and so the trajectories) in Vyax(A, im B, XK) (or in XK) that can be generated by the state Equation 
5.7, when x(t) € Vmax(A, im B, K), is identical to those generated by x(t) = (A+ BF)x(t)+ BLu(t) for a 
matrix L satisfying im BL = im BN Vmax(A, im B, K) and the admissible inputs u(t). Then, by Equation 
5.11, Vreach(K) = R(A + BF, im BN Vmax (K)), where Vinax (KC) = Vmax(A, im B, KX). Clearly, Vyeacn(K) D 
im BM Vmax (KC) and the subspace Vyeqc; (KC) is the minimum (A + BF)-invariant satisfying this inclusion 
(see the remark immediately after Theorem 5.1), that is, it is the minimum (A, im B)-controlled invariant 
self-bounded with respect to K. Thus, we have 


Theorem 5.8: 


Let F be a real matrix satisfying (A + BF)Vmax(K) C Vmax(K), where Vinax (KC) = Vax (A, im B, K). Then, 
Vreach(K) = R(A + BF, im BN Vmax(K)) = Vsp,min(A, im B, K). (5.30) 


Dual to the above results we have 
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Definition 5.5: 


An (A, ker C)-conditioned invariant S containing T is said to be self-hidden with respect to LT if 


S C Smin(A, ker C,Z) + ker C. (5.31) 


Lemma 5.13: 


Let G be ann x p real matrix satisfying 
(A+ GC)Smin(A, ker C,Z) C Smin(A, ker C, Z). (5.32) 
Then, every (A, ker C)-conditioned invariant S (containing T) self-hidden with respect to L satisfies 
(A+ GC)S CS. (5.33) 
If (A, ker C)-conditioned invariants S; and S2 containing Z are self-hidden with respect to Z, so is Sj + 
S2. Therefore, the above set has a maximum element which is called the maximum self-hidden (A, ker C)- 


conditioned invariant with respect to T denoted by Ssh max (A, ker C, Z). The subspace Ssp max (A, ker C, Z) 
is related to Vinax(A, Z, ker C) and Sin(A, ker C, Z) as follows. 


Theorem 5.9: 


Ssh.max(A; ker C, Z) = Smin(A, ker C, Z) + Vax (A; Z, ker C). (5.34) 


Furthermore, we have 


Theorem 5.10: 


Sshmax(A; ker C,Z) = (ker C+ Smin(Z))N (A+ GC)! (ker C + Smin(Z)) 

N---(A+.GC)%Y (ker C + Smin(Z)), (5.35) 

where Smin(Z) := Smin(A, ker C, Z) and the matrix G satisfies Equation 5.32, that is, (A + GC)Smin(Z) C 

Smin(Z). 

Remark 

The right-hand side of Equation 5.35 represents the largest A+ GC invariant subspace contained in 

ker C+ Smin(Z). 

Remark 


Equation 5.35 indicates that S.)max(A, ker C,Z) represents the set of unobservable states through a 
dynamic observer when unknown disturbances are present if Z represents the disturbance channel. See 
Basile and Marro (1992, Section 4.1.3) for details on the topic. 


5.7 Internal, External Stabilizability 


We now introduce the notions of stability associated with controlled and conditioned invariants. 
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Definition 5.6: 


An (A, im B)-controlled invariant V is said to be internally stabilizable if, for any initial state xp € V, 
there is a control u(t) such that x(t) € V for all t > 0 and x(t) converges to the zero state as t > 0x, or, 
alternatively (and, in fact, equivalently), if there exists an m x n real matrix F satisfying (A+ BF)V CV 
and (A+ BF)| VY is stable. 


Definition 5.7: 


An (A, im B)-controlled invariant V is said to be externally stabilizable if, for any initial state x) € X, there 
is a control u(t) such that x(t) converges to V as t + 00, or, alternatively (and, equivalently), if there exists 
an m x n real matrix F satisfying (A+ BF)V C V and (A+ BF)| 4’/V is stable. 


Internal and external stabilizabilities can be easily examined by applying appropriate coordinate trans- 
formations (consisting of new sets of basis vectors) for the state and input spaces. For this, let V be an (A, 
im B)-controlled invariant. Also, let n,; := dim Vyeacn(V), ny := dim V, and ns := dim Smin(A, V, im B). 
Define n x n and m x m nonsingular matrices T=[T, T2 T3 Tg] and U =[U; U2], respectively, as 
follows: Choose Uj, T;, and T>, so that im BU] = VN imB, im Ty = Vyege,(V), and im [Ti T2| =yV. 
Also, choose U2, so that im B[U, U2] = im B. Noting that Veg-n(V) = VN Smin(A, V, im B) (see Equa- 
tions 5.27 and 5.30; let K = V), select T3 satisfying im [T T3] = Smin(A, V, im B) with im BU2 C im T3. 
Then, we have 7 7 : 7 

An Ai Ais Ala 
AVS TAT lk, Jee eS ee de (5.36) 


By, =O 
s 0 0 
B:=T BU = Soa 5.37 
on 2B (5.37) 
0 0 
Here, A is divided into blocks in accordance with the ranks of T;, T2, T3, Ta, that is, Aj], Ay,...,Aa4 
respectively have dimensions nj x 1, nN, X M2,...,N4 X n4, where ny := rankT, = n,, nq := rankT 
ny —ny,..., and B is divided similarly, for example, B; and B32 have dimensions n, x m, and n3 x m2, 


where m, and mz are, respectively, the numbers of columns of U; and U2. Note that the zero matrix in the 
second block row of A is due to the following facts: V;eqc,(V) is (A + BF)-invariant for every F satisfying 
(A+ BF)Y C Y, but the second block row of B is zero (which makes F ineffective in the block). The fourth 
block row of A has zero blocks since V is (A + BF)-invariant for any F defined above, but the fourth block 
row of B is zero. Now, using F satisfying (A + BF)V C V, consider A + BF with respect to the new bases. 
Noting that A3, + B32Fo) = 0 and Az) + B39 Fx = 0 by construction, we obtain 


Aut BuFu At Buk Ai3t+Bukis AwtBuFi 


3 gis = 0 A A23 Ar 
A+BF=T (A+ BF)T = he aS Dy Oe (5.38) 
0 0 A33 + B32F23 Aza + Bs2Fa 


where : 2 4 7 
= Fy, Fyn Fy3 a x 4 i = 7 -1 
F=|- 7 s ~ |, A:=T AT, B:=T BU, and F:=U “FT. 
B Fy, Fy3 Pha 
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Note that Az cannot be altered by any linear state feedback satisfying (A + BF)V C Y, that is, the 
(2,2)-block of A+ BF remains A») for any real F satisfying 


In, 0 In, 0 
~~ ww 0 XL 0 LI 
(A+ BF)im ” ) Cim : ‘4 ; (5.39) 
0 0 0 0 
where 0’s are zero matrices with suitable dimensions. Thus, 
Lemma 5.14: 
o((A + BF)|V/Vreach(V)) = (Az) (5.40) 


(i.e., fixed) for all F satisfying (A+ BF)V C V. Here, (A+ BF)|V/Vyeach(V) is the induced map of A+BF 
restricted to V/Vyeach(V). 


Note also that by construction the pair (Aj1, B11) is reachable. Hence, by Theorem 5.2, we have 


Lemma 5.15: 


o((A + BF)| Vreach(V)) = 0(Aq; + By Fy) can be freely assigned by an appropriate choice of F satisfying 
(A+ BF)V CY (or F satisfying Equation 5.39). 


The eigenvalues of (A+ BF)|V/Vyeacn(V) are called the internal unassignable eigenvalues of V. 
The internal unassignable eigenvalues of Vmax (A, im B, ker C) are called invariant zero’s of the system 
&:=[A, B, C] (or triple (A, B, C)), which are equal to the transmission zero’s of C(sI—A)7!B if (A, B) is 
reachable and (A, C) is observable. Table 5.1 shows how freely the eigenvalues can be assigned for A + BF 
by choosing F satisfying (A + BF)V C VY given an (A, im B)-controlled invariant. Theorems 5.11 and 5.12 
given below easily follow from the above lemmas and Table 5.1. 


Theorem 5.11: 


An (A, im B)-controlled invariant V is internally stabilizable if and only if all its internal unassignable 
eigenvalues have negative real parts. 


TABLE 5.1 Spectral Assignability of (A + BF) | W, Given an (A, im B)-Controlled Invariant V 
Ww X/(V + Vreach) (V+ Vreach)/V V/Vreach (V) Vreach(V) 
Assignability Fixed Free Fixed Free 


Note: The table indicates that o((A + BF) | ¥/(V + Vyeach)) is fixed for all F satisfying (A + BF)V C 
V,0((A+ BF) | (V+ Vyeach)/V) is freely assignable (up to a symmetric set) by choosing an 
appropriate F satisfying (A + BF)V C YV, and so on. 
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Theorem 5.12: 


An (A, im B)-controlled invariant V is externally stabilizable if and only if V + Vreach is externally stable. 


Remark 
V+Vyeach is A-invariant since A(V + Vreach) = AV + AV reach = V tim B+ Vyeach = V+ Vreach: 


Lemma 5.16: 


If the pair (A, B) is stabilizable, then all (A, im B)-controlled invariants are externally stabilizable. 


Remark 


The matrix F can be defined independently on V and on 4’/V. 
Dual to the above internal and external stabilizabilities for a controlled invariant are, respectively, 
external and internal stabilizabilities for a conditioned invariant which will be defined below. 


Definition 5.8: 


An (A, ker C)-conditioned invariant S is said to be externally stabilizable if there exists an n x p real matrix 
G satisfying (A+ GC)S C S and (A+ GC)| ¥/S is stable. 


Definition 5.9: 


An (A, ker C)-conditioned invariant S is said to be internally stabilizable if there exists ann x p real matrix 
G satisfying (A+ GC)S C S and (A+ GC)|S is stable. 


How freely the eigenvalues can be chosen for A + GC by means of G is given in Table 5.2, from which 


necessary and sufficient conditions of the stabilizabilities follow easily. (See Basile and Marro (1992) for 
further discussions on the topic of the internal and external stabilizabilities.) 


5.8 Disturbance Localization (or Decoupling) Problem 


One of the first problems to which the geometric notions were applied is the problem of disturbance 
localization (or disturbance decoupling). As we see below, the solution to the problem is remarkably 
simple in geometric terms. The solution and analysis of this problem can also be used to solve other 


TABLE 5.2 Spectral Assignability of (A + GC) | W, Given an (A, ker C)-Conditioned Invariant S 
Ww X/S sh max (A ker C, S) Ssh,max (A, ker C,S)/S S/S OS ynobs SO Synobs 


Assignability Free Fixed Free Fixed 


Note: The table indicates that o((A + BF)|V/S. max (A, ker C, S)) is freely assignable (up to a symmetric set) by choosing 
an appropriate G satisfying (A + GC)S C S,o((A + BF)|Ssh, max (A, ker C, S)/S) is fixed for all G satisfying (A + 
GC)S C S, and so on. 
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more involved problems (e.g., model following, decoupling, and disturbance decoupling in decentralized 
systems). 
We will be concerned with a time-invariant linear control system Xg:=[A, B, D, C] described by 


x(t) = Ax(t) + Bu(t) + Dw(t), x(0) =x», (5.41) 
y(t) = Cx(t), (5.42) 
where the column vectors x(t) € ¥ := R", u(t) € R™, y(t) € RP, and w(t) € R™4 are, respectively, the state, 
input, output, and unknown disturbance of the system at time t > 0, the vector x, € R” is the initial state 
and the real matrices A, B, C, and D have consistent dimensions. We assume that u(t) and w(t) are 


piecewise continuous. Here, w(t) is called a “disturbance” and it can neither be measured nor controlled. 
The above notation will be standard in the sequel. If we apply the linear state feedback control law 


u(t) = Fx(t) (5.43) 
to the above system, we obtain the state equation 
x(t) = (A+ BF)x(t)+ Dw(t), (0) = xo; (5.44) 


where F is an mx n real matrix. Our problem is to choose the control law (Equation 5.43) so that 
the disturbance doses not affect the output in the resulting closed-loop system given by Equations 5.44 
and 5.42, that is, so that y(t) = 0 for all t > 0 for any w(t), t > 0, provided x(0) = x, = 0. By virtue of 
Theorem 5.1, note that at any time t > 0 all the possible states due to all the admissible w(t), 0< t<t 
are characterized by 


R(A+BF, imD):=imD+(A+BF)imD+---+(A+BF)"! imD. (5.45) 


Therefore, in algebraic terms, the above problem can be restated as 


Problem 5.1: Disturbance Localization (or Disturbance Decoupling) Problem 


Givenn x n,n x m,n x mg, and p x n real matrices A, B, D, and C, find an m x n real matrix F satisfying 
R(A+ BF, imD) C ker C. (5.46) 


It is not always possible to find F satisfying Inclusion 5.46. The following theorem gives the necessary 
and sufficient condition for the existence of such an F in terms of given matrices. (See Basile and Marro 
(1992, Section 4.2) and Wonham (1985, Section 4.3).) 


Theorem 5.13: 


There exists an m x n real matrix F satisfying Inclusion 5.46 if and only if 


im D C Vimax(A, im B, ker C) (5.47) 


Proof. [Only if part.] Trivially, imDC R(A+BF,imD). Since R(A+ BF,imD) is an (A+ BF)- 
invariant, it is an (A, im B)-controlled invariant in view of Theorem 5.5, and, by assumption, is contained 
in ker C. Therefore, R(A + BF, im D) C Vmax(A, im B, ker C). Thus, Equation 5.47 holds. 
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[If part] Let F be such that 
(A + BF)Vmax(A, im B, ker C) C Vinax (A, im B, ker C). (5.48) 


Inclusion 5.47 implies, consecutively, (A + BF)im D C Vmax(A, im B, ker C),...,(A + BF)"*“limDc 
Vmax (A, im B, ker C). Therefore, R(A + BF, im D) C Vmax(A, im B, ker C) C ker C. QED 


Remark 


Let n* := dim Vmax (A, im B, ker C), and let T be a real nonsingular matrix for which the first n* columns 
form a basis of Vmax(A, im B, ker C). Then, Inclusion 5.47 means that, with an m x n real matrix F 
satisfying Inclusion 5.48, the coordinate transformation x(t) = Tx(t) transforms Equations 5.44 and 5.42 


to 
xa(t)]  [Aut+BiFi Av+BiF] fat) Dy 
Ea 7 | 0 Ax tee eed i | 0 w(t) (5.49) 
and 
w=[0 G] be ‘ (5.50) 


where [Fi Fy] = FT. Clearly, w(t) does not affect x(t); hence, no effect on y(t). 


5.9 Disturbance Localization with Stability 


In the previous section, the disturbance localization problem without additional constraints was solved. 
In this section, the problem is examined with an important constraint of stability. For the system 
Xg:=[A,B,D,C] described by Equations 5.44 and 5.42, or for the matrices A, B, D, and C, we have the 
following. 


Problem 5.2: Disturbance Localization Problem with Stability 


Find (if possible) an m x n real matrix F, such that (1) Inclusion 5.46 holds and (2) A+ BF is stable, that is, 
the eigenvalues of A + BF have negative real parts. 


Trivially, for condition (2) to be true, it is necessary that the pair (A, B) is stabilizable. We have (see 
Basile and Marro, 1992, Section 4.2) 


Theorem 5.14: 


Let A,B,D, and C be as before, and assume that the pair (A,B) is stabilizable. Then, the distur- 
bance localization problem with stability has a solution if and only if (1) Inclusion 5.47 holds and (2) 
V<p,min (A, im B+ im D, ker C) is an internally stabilizable (A, im B)-controlled invariant. 


Remark 


An alternative condition can be found in Wonham (1985, Theorem 5.8). 
We will first prove that the above two conditions are sufficient. This part of the proof leads to a 
constructive procedure to find a matrix F which solves the problem. 
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Sufficiency proof of Theorem 5.14. Thecondition (2) means that there isa real matrix F, such that (A + BF) 
Vepmin(A, im B+ im D, ker C) C Vey min(A, im B+imD,kerC) and Vy min(A,imB+imD,kerC) is 
internally stable. Since the pair (A, B) is stabilizable, V.p min(A, im B+ im D, ker C) is externally stabi- 
lizable by virtue of Lemma 5.16. Since the internal stability of V,p min(A, im B + im D, ker C) (with respect 
to A+BF) is determined by F| V,p,min(A, im B + im D, ker C) and since the external stability depends only 
on F| X/V.p.min(A, im B+ im D, ker C), the matrix F can be chosen so that the controlled invariant is 
both internally and externally stable, hence, A + BF is stable. It remains to show that Equation 5.46 holds. 
Since V.p min(A, im B + im D, ker C) is self-bounded with respect to ker C, and since Equation 5.47 holds, 
we have 


Vsb,min (A, im B+ im D, ker C) D (im B+ im D)N Vmax (A, im B+ im D, ker C) 
> (im B+ im D) NM Vax (A, im B, ker C) 
= (im BN Vmax (A, im B, ker C)) + im DD im D. 


This inclusion and the (A + BF)-invariance of V;p min(A, im B+ im D, ker C) imply R(A + BF, im D) C 
Vep,min (A, im B+ im D, ker C) C ker C. QED 


Remark 


In the remark after Theorem 5.13, redefine n* := dim Vip min(A, im B+ im D, ker C). Then, o(A+ 
BF|V;5,min(A; im B + im D, ker C)) =o (An + Bhi) and o(A + BP|A'/Vsp, min(A, im B+ im D,ker C)) = 
9 (Az, + Boh). Therefore, we choose F, and F, so that Aj; + BF; and Ay. + BF) are stable and 
A>, + BF; = 0. Then, the desired F is given by F= [Fi F] To}, 
To prove the necessity we use the following 


Lemma 5.17: 


If there exists an internally stabilizable (A, im B)-controlled invariant V satisfying V C ker CandimD CV, 
then V.b min(A; im B+ im D, ker C) is an internally stabilizable (A, im B)-controlled invariant, that is, it is 
(A+ BF)-invariant and A+ BF| Vp, min(A, im B+ im D, ker C) is stable for some real matrix F. 


Proof. The lemma follows from Schumacher (1983, Propositions 3.1 and 4.1). It is also an immediate 
consequence of Lemma 4.2.1 in Basile and Marro (1992). 


Necessity proof of Theorem 5.14. Suppose that the problem has a solution, that is, there is a real matrix F 
such that A+ BF is stable and V := R(A + BF, im D) C ker C. Clearly, im D C YV. Also, V is an (A + BF)- 
invariant contained in ker C. Hence, V C Vmax(A, im B, ker C). Therefore, condition (i) holds. To show 
condition (ii), note that V is internally (and externally) stable with respect to A+ BF, V C ker C, and 
im D C V. Condition (ii) follows by Lemma 5.17. QED 


5.10 Disturbance Localization by Dynamic Compensator 


In the previous section, we used the (static) state-feedback control law u(t) = Fx(t) to achieve disturbance 
rejection and stability. In this section, we use measurement output feedback with a dynamic compensator 
&- placed in the feedback loop to achieve the same objectives. 

More specifically, we are concerned with the system gy, := [A, B, D, C, Cineas] described by 


x(t) = Ax(t)+ Bu(t)+ Dwi(t), x(0)=xo (5.41) 
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y(t) = Cx(t), (5.42) 
Ymeas(t) = Creasx(t), (5.51) 


where Ymeas(t) € Vineas = R?™ stands for a measurement output and Cyyeqs is a Pm X n real matrix. Define 
the dynamic compensator &, := [A¢, Be, C,, K;] by 


X(t) = AcXc(t) + BeYmeas(t), x-(0) = Xco» (5.52) 
u(t) = C-x¢(t) + KeYmeas(t), (5.53) 


where x¢(t), Xco € Xe = R", and A;, B,, C,, and K; are, respectively, n- X Ne, M%e X Pm» M X Ne, MX Pm real 
matrices. We wish to determine A;, B,, C,, and K; such that (1) y(t) = 0 for all t >0 for any (admissible) 
w(t), t > 0 provided xo = 0 and x;o = 0, and (2) x(t) and x,(t) converges to zero as t > +00 forall x, € ¥ 
and x¢o € %¢. For this, it is convenient to introduce the extended state defined by 


fetes | ek X xX, =R tM, 
Ke 


where x € ¥ and x, € X;. Then, the overall system (including Xg,, and X,) can be rewritten as 
5(t) = Ax(t) + Dwit), %(0) = So, (5.54) 
y(t) = CX(t), (5.55) 


where % :=[x/,  x/ ii and the matrices are defined by 


co 

A A+ BK; Cmeas BC, rn D - 

A:= » Di= » Ci=|C Oj. 5.56 
BeCmeas Ac | ( 


Remark 
Define 


~ [Ke C, 
Kei= [5 mal (5.57) 
Then, it is easy to verify that 
A=A,+BoR-Cmeas- (5.58) 


(See Basile and Marro (1992, Section 5.1) and Willems and Commault (1981).) 


Remark 


An important special case of &, is a state estimate feedback through an observer. More specifically, let Ly 
and Lysatisfy 


L1 Cineas + Ly = In (5.59) 


and consider an asymptotic observer (or state estimator) described by 


X(t) = Ax¢(t) + Bu(t) + G{CreasXc(t) —Ymeas(t)}, x,-(0) = Xco» (5.60) 
Xest(t) = Lox¢(t) + LiyYmeas(t), (5.61) 
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where Xes+(t) € R” is an “estimate” of x(t) (under proper conditions) and G, L;, and L, are real matrices 
with appropriate dimensions. If we use a state estimate feedback law 


u(t) = Fxes¢(t) (5.62) 


with an m x n real matrix F, then the overall system reduces to Equations 5.54 and 5.55, where A, D, and 
C are given, instead of Equation 5.56, by 


= ae eae A+ ihe me eee lo) re sie) 

It should also be noted that, if we apply the coordinate transformation 

ale ae 
I, —In 
Equations 5.54 and 5.63 can be rewritten as 

x(t) = Ax(t) + Dw(t), (0) = Xo, (5.64) 

a A+ BF BFL D 
As=| - a Ds i (5.65) 


from which it is apparent that F and G can be selected independently (based on Theorem 5.2 and its dual) 
to make A stable, that is, the separation property holds. (See Basile and Marro (1992, Sections 3.4.2, 5.1.2) 
for further discussions.) 

With the above notation the problem can be restated in geometric terms as: 


Problem 5.3: Disturbance Localization Problem with Stability 
by Dynamic Compensator 


Find (if possible) a number n,(= dim %,) and real matrices A,, B,,C;, and K, of dimensions ne X Me, Nc X 
Pm ™ X Ne, and m X pm, respectively, such that 


(i) R(A,im D) Cc ker C, (5.66) 
and 


(ii) A is stable. (5.67) 


Remark 


Noting that R(A, im D) is the minimum A-invariant containing im D, it is easy to see that condition (i) 
is equivalent to the following condition: 
(i)’ there is an A-invariant V satisfying 


imD CV CkerC. (5.68) 


The conditions under which the above problem is solvable are given by the following theorems. 


Theorem 5.15: 


Let (A, B) be stabilizable, and also let (A, Cmeas) be detectable. Then, the disturbance localization problem 
with stability by dynamic compensator has a solution if and only if there exists an internally stabilizable 
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(A, im B)-controlled invariant V and an externally stabilizable (A, ker Cyneqs)-conditioned invariant S 
satisfying the condition 


imDCSCVCkerC. (5.69) 
Proof. See Basile and Marro (1992, Section 5.2) and Willems and Commault (1981). QED 


The above condition is an existence condition and it is not convenient for testing. The following 
theorem (see Theorem 5.2-2 in Basile and Marro, 1992) gives constructive conditions and they can be 
readily tested. 


Theorem 5.16: 


Assume that (A, B) is stabilizable and (A, Cineas) is detectable. The disturbance localization problem with 
stability by dynamic compensator has a solution if and only if the following conditions hold: 


1. 
Smin(A, ker Cmeqs, im D) C Vmax (A, im B, ker C), (5.70) 


ii, Smin(A, ker Creas, im D) + Vmax(A, im D, ker CN ker Creas) is an externally _ stabilizable 
(A, ker Cyneqs )-conditioned invariant 

iii, V.pmin(A, im B+ im D, ker C) + Vmax(A, im D, ker CN ker Cyneas) is an internally stabilizable 
(A, im B)-controlled invariant. 


Remark 


The subspaces Smin(A, ker Creas,im D) + Vmax(A, im D, ker CN ker Cyneas) and Vp min(A, im B+ 
im D, ker C) + Vmax(A, im D, ker CN ker Creas) defined above are, respectively, an (A, ker Cyyeas)- 
conditioned invariant and an (A, im B)-controlled invariant. (See Lemma 5.18 at the end of this section 
for the proof the above remark.) 


Proof of Necessity of Theorem 5.16. See Basile and Marro (1992, Section 5.2) and Basile et al. (1986). 


QED 

Proof of Sufficiency for Theorem 5.16. Let 
S := Smin(A, ker Cregs, im D) + Vax (A, im D, ker CN ker Cyeas), (5.71) 
V := Vsp,min(A, im B + im D, ker C) + Vmax(A, im D, ker CM ker Cineas). (5.72) 


We will show that the above S and Y satisfy the conditions of Theorem 5.15. By assumptions (ii) 
and (iii), S is externally stabilizable and V is internally stabilizable. By Inclusion 5.70 and Equation 5.71, 
trivially im D C S. By Equation 5.72, V C ker C trivially also. To see the validity of the middle inclu- 
sion of Inclusion 5.69, that is, S C Y, it suffices to show Smin(A, ker Creas, im D) C Vop,min(A, im B+ 
im D, ker C). For this, first observe that, by Inclusion 5.70, we have im D C Vmax(A, im B, ker C), which 
implies Vinax(A, im B+ im D, ker C) = Vmax (A, im B, ker C) since trivially Vinax (A, im B+ im D, ker C) D 
Vmax(A, im B, ker C) and AVmax(A, im B + im D, ker C) C Vmax (A, im B+ im D, ker C) + im B+im D= 
Vmax (A, im B+ im D, ker C) + im B. In addition, Inclusion 5.70 implies Smin(A, ker Cyeqs, im D) C ker C, 
which leads to Smin(A, ker Cyeqs, im D) = Smin(A, ker Cregs M ker C, im D) C Spin(A, ker C, im D) C 
Smin(A, ker C, im B+ im D). 
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By using Inclusion 5.70, the above facts, and Theorem 5.7, we have 


Smin(A, ker Creass im D) C Vmax(A, im B, ker C) N Smin(A, ker C, im B + im D) 
= Vinax(A, im B+ im D, ker C) Smin(A, ker C, im B + im D) (5.73) 
= sb,min (A, im B+ im D, ker C). 


Thus, Inclusion 5.69 holds. QED 
Procedure for Finding A,, Bc, Cc, and Ke 
Define S and V by Equations 5.71 and 5.72. Let £ be a subspace of R” satisfying 
L® (SN ker Cmeas) = S. (5.74) 


Clearly, 
LO ker Creqs = {0}. (5.75) 


Now let Leomp be a subspace satisfying LB Leomp = R” and Leomp D ker Cimeqs, and define the projec- 
tion Lz on Leomp along L. Select L; such that 


L1Cineas = In — L2. (5.76) 


Such an Ly exists since ker(I, — L2) = Leomp D ker Cineas- 
Choose real matrices F and G so that 


(A+BF)V CY, (5.77) 
(A Te GC neas)S cS, (5.78) 


and A+ BF and A+ GCyneqs are stable. Then, use Equations 5.60 through 5.62 as a dynamic compensator, 
or equivalently, set A, := A+ BFL2 + GCyeqs, Be := BFL, — G, C;:=FL2 and K,:=FL, in Equations 5.52 
and 5.53. From Equation 5.65, the overall system is clearly stable (and so is A). It can also be shown (see 
Basile and Marro, 1992; Lemmas 5.1.1 and 5.1.2, and Section 5.2.1) that 


b= {| . |iversesl 
v—s 


is A-invariant and satisfies Equation 5.68. 


Lemma 5.18: 


The subspaces S defined by Equations 5.71 is an (A, ker Cieas)-conditioned invariant; and, if Inclusion 5.70 
holds, then V defined by Equation 5.72 is an (A, im B)-controlled invariant. 


Proof. Since Vmax(A, im D, ker CM ker Cineas) C ker Cineas, by the last equality of Lemma 5.2 we have 
SO ker Creas = Smin(A, ker Cyneas, im D) N ker Creas + Vmax(A, im D, ker CN ker Cineas). Then, it is easy 
to see A(S N ker Cineas) C S + im D. But, im D C Smin(A, ker Creqs, im D) C S. Thus, A(S N ker Cireas) C 
S. To prove that V is an (A, im B)-controlled invariant, first it is easy to show AV CV+imB+imD. 
But, by Inclusion 5.73, im D C Vey. min(A, im B+ im D, ker C) C V. Thus, AV C V+ im B. QED 
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5.11 Conclusion 


An introduction to the geometric method (often called geometric approach) has been provided. The basic 
tools have been described and applied to the disturbance localization (also called disturbance decouping) 
problems to demonstrate the use of the tools. Numerous research articles were published on the subject 
for the last 40 years. Some are included in the list of references in this chapter and a great many others can 
be found in the books and articles already referenced in this chapter. Reports for practical applications 
can also be found in the literature, for example, Takamatsu et al. (1979); Massoumnia et al. (1989); 
Prattichizzott et al. (1997); Barbagli et al. (1998); and Marro and Zattoni (2004). This chapter has dealt 
with continuous-time systems. Many articles in the literature treat discrete-time systems, for example, 
Akashi and Imai (1979), Hamano and Basile (1983), and Marro and Zattoni (2006). 

The notion of controlled invariant subspaces has been extended to “almost” controlled invariant 
subspaces, to which the state trajectory can stay arbitrary close by a suitable choice of inputs, possibly 
with high-gain feedback (Willems, 1981). The theory involves use of distributions for inputs. The dual 
notion of “almost” conditionally invariant subspaces has also been introduced (Willems, 1982). The 
geometric approach has also been extended to more general or different classes of linear systems, for 
example, 2D systems (Conte and Perdon, 1988), systems over a ring including delay differential systems 
(Conte and Perdon, 1998), and periodic systems (Longhi and Monteriu, 2007). The geometric notions for 
linear systems have been extended to nonlinear systems (controlled invariant distributions, in particular) 
using differential geometric tools and problems such as disturbance localization and output decoupling 
have been studied in the literature (see Nijmeijer and van der Schaft (1990), Isidori (1995), and Mattone 
and De Luca (2006), for instance). A 40-year historical perspective of the development of geometric 
methods for linear and nonlinear systems with an insight into the future can be found in Marro (2008). 
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6.1 Introduction 


For control system design, it is useful to characterize multi-input, multi-output, time-invariant linear 
systems in terms of their transfer function matrices. The transfer function matrix of a real m-input, 
p-output continuous-time system is a (p x m) matrix-valued function G(s), where s is the Laplace trans- 
form variable; the corresponding object in discrete time is a (p x m) matrix-valued function G(z), where 
z is the z-transform variable. Things are particularly interesting when G(s) or G(z) is a proper rational 
matrix function of s or z, that is, when every entry in G(s) or G(z) is a ratio of two real polynomials in s 
or z whose denominator’s degree is at least as large as its numerator’s degree. In this case, the system has 
state space realizations of the form 


x(t) = Ax(t) + Bu(t) 
y(t) = Cx(t) + Du(t) 


or 


x(k +1) = Ax(k) + Bu(k) 
y(k) = Cx(k) + Du(k), 


where the state vector x takes values in R”. Any such realization defines a decomposition G(s) = 
C(sI, —A)~'B+D or G(z) = C(zI, — A)~'B+D for the system’s transfer function matrix. A realiza- 
tion is minimal when the state vector dimension n is as small as it can be; the MacMillan degree 8y4(G(s)) 
or 8y4(G(z)) is the value of 1 in a minimal realization. A system’s MacMillan degree is a natural candidate 
for the “order of the system.” 

It is not easy to construct minimal realizations for a multi-input, multi-output system with an arbitrary 
proper rational transfer function matrix; even determining such a system’s MacMillan degree requires 
some effort. Circumstances are simpler for single-input, single-output (SISO) systems. Consider, for 
example, a real SISO continuous-time system with proper rational transfer function g(s). We can express 
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g(s) asa ratio 
g(s)= ae) (6.1) 


where q(s) is a polynomial of degree n, p(s) is a polynomial of degree at most n, and p(s) and q(s) are 
coprime, which is to say that their only common factors are nonzero real numbers. The coprimeness of 
p(s) and q(s) endows the fractional representation Equation 6.1 with a kind of irreducibility; furthermore, 
Equation 6.1 is “minimal” in the sense that any other factorization g(s) = p(s)/q(s) features a denominator 
polynomial q(s) whose degree is at least n. 

The MacMillan degree 5y(g(s)) is precisely n, the degree of q(s) in Equation 6.1. To see that n < 
5u(g(s)), suppose (A, B, C, D) are the matrices in a minimal realization for g(s). Set i = 8u(g(s)), so A 
is (” x n). The matrix (sI;, — A)~! has rational entries whose denominator polynomials have degrees at 
most fi. Multiplying on the left by C and on the right by B and finally adding D results in a rational 
function whose denominator degree in lowest terms is at most #1. This rational function, however, is g(s), 
whose lowest-terms denominator degree is n, whence it follows that n < fn. 

To finish showing that 8y(g(s)) =n, it suffices to construct a realization whose A-matrix is (n x n). 
There are many ways to do this; here is one. Begin by setting d = lim), oo g(s); d is well-defined because 
g(s) is proper. Then 

B(s) 
g(s) © +d, 


where the degree of p(s) is at most n — 1. Cancel the coefficient of s” from q(s) and p(s) so that 


q(s) = s" + qis" | +++++qn—18+ Gn 


and 
pls) =yis" | + yas"? +--+ + Yn-1S+ Yn 


It is not hard to verify that g(s) = C(sIy — A)~!B+ D when 


0 1 0 0 0 

0 0 1 0 0 0 

0 0 0 1 0 
A= ge! [ig BS ce|4 (6.2) 

1 0 
0 0 0 1 0 
—n —n-1 —qz2 —q) 1 
and 

Cave She ae ale DS (6.3) 


To summarize the foregoing discussion, if g(s) is the transfer function of a real time-invariant 
continuous-time SISO linear system, and g(s) is proper rational, then g(s) possesses a fractional 
representation Equation 6.1 that 


is irreducible because p(s) and q(s) are coprime, or, equivalently. 
¢ is minimal because the degree of q(s) is as small as it can be in such a representation. 


Moreover, 


e the degree of q(s) in any minimal and irreducible representation Equation 6.1 is equal to the 
MacMillan degree of g(s). 
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Replacing s with z results in identical assertions about discrete-time SISO systems. 

Our goal in what follows will be to generalize these results to cover continuous-time, multi-input, multi- 
output (MIMO) systems with (p x m) proper rational transfer function matrices G(s). The development 
is purely algebraic and applies equally well to discrete-time MIMO systems with proper rational transfer 
function matrices G(z); simply replace s with z throughout. In Section 6.2, we present MIMO versions of 
the fractional representation Equation 6.1, complete with appropriate matrix analogues of irreducibility 
and minimality. In Section 6.3, we relate the results of Section 6.2 to the MacMillan degree of G(s). We 
obtain as a bonus a minimal realization of G(s) that turns out to be the MIMO analogue of the realization 
in Equations 6.2 and 6.3. In Section 6.4 we prove a well-known formula for the MacMillan degree of 
G(s) and discuss briefly some connections between the material in Section 6.2 and the theory of ARMA 
models for MIMO systems, with a nod toward some modern results about stable coprime factorization 
and robust control. The algebra we will be employing throughout, while tedious at times, is never terribly 
abstract, and much of it is interesting in its own right. 


6.2 Polynomial Matrix Fraction Descriptions 


We begin by establishing some basic terminology. A real polynomial matrix is a matrix each of whose 
entries is a polynomial in s with real coefficients. We often omit the adjective “real” but assume, unless 
stated otherwise, that all polynomial matrices are real. One can perform elementary operations on poly- 
nomial matrices such as addition and multiplication using the rules of polynomial algebra along with 
the standard formulas that apply to matrices of numbers. Likewise, one can compute the determinant 
det F(s) of a square polynomial matrix F(s) by any of the usual formulas or procedures. All the familiar 
properties of the determinant hold for polynomial matrices, for example, the product rule 


det[ Fj (s)F2(s)] = [det F)(s)][det F2(s)], (6.4) 


which applies when Fj (s) and F(s) are the same size. 
There are two shades of invertibility for square polynomial matrices. 


Definition 6.1: 


A square polynomial matrix F(s) is said to be nonsingular if, and only if, det F(s) is a nonzero polynomial 
and unimodular if, and only if, det F(s) is a nonzero real number. 


Thus if F(s) is nonsingular, we are free to compute F~!(s) using the adjugate-determinant formula for 
the inverse [12], namely, 


Fol(s)= adjF(s). 


1 
det F(s) 

The (i,j) entry of adjF(s) is (— 1)'+/ times the determinant of the matrix obtained from F(s) by elim- 
inating its jth row and ith column. F-1(s) is in general a rational matrix function of s. If F(s) is not 
just nonsingular but also unimodular, then F—1(s) is actually a polynomial matrix; thus unimodular 
polynomial matrices are “polynomially invertible.” 

From Definition 6.1 along with the product rule Equation 6.4, the product of two nonsingular poly- 
nomial matrices is nonsingular and the product of two unimodular polynomial matrices is unimodular. 
Furthermore, we have the following “pointwise” characterization of unimodularity: a square polynomial 
matrix F(s) is unimodular precisely when F(s,) is an invertible matrix of complex numbers for every 
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complex number sy. This statement follows from the fundamental theorem of algebra, which implies that 
if det F(s) is a polynomial of nonzero degree, then det F(s,) = 0 for at least one complex number so. 
As an example, consider the two polynomial matrices 


s+2 s+2 s+t2 s+4 
Fy(s) = kee ae and F)(s)= | ue] ; 
F,(s) is unimodular and F;(s) is merely nonsingular; in fact, 
1 —l1 
= z l/s+2 -s—2 
1 1 
FL (s)= = ; and F, @=7] : a 

s+ 


Weare ready to define the matrix generalization(s) of the fractional representation Equation 6.1. 


Definition 6.2: 


Let G(s) be a real (p x m) proper rational matrix function of s. 


« A right matrix fraction description, or right MFD, of G(s) is a factorization of the form G(s) = 
P(s)Q71(s), where P(s) is a real (p x m) polynomial matrix and Q(s) is a real (m x m) nonsingular 
polynomial matrix. 

« A left matrix fraction description, or left MFD, of G(s) is a factorization of the form G(s) = 
Qr\(s)Px(s); where P,(s) is a real (p x m) polynomial matrix and Qr(s) is a real (p x p) nonsingular 
polynomial matrix. 


It is easy to construct left and right matrix fraction descriptions for a given (p x m) proper rational 
matrix G(s). For instance, let q(s) be the lowest common denominator of the entries in G(s). Set P(s) = 
q(s)G(s); then P(s) is a polynomial matrix. Setting Q(s) = q(s)Im makes P(s)Q7\(s)a right MED for G(s); 
setting Qz(s) = q(s)Ip makes Qr'(s)P(s) a left MFD of G(s). 

Next we introduce matrix versions of the notions of irreducibility and minimality associated with the 
fractional representation Equation 6.1. First consider minimality. Associated with each right (or left) 
MED of G(s) is the “denominator matrix” Q(s) (or Qr(s)); the degree of the nonzero polynomial det Q(s) 
(or det Q;(s)) plays a role analogous to that of the degree of the polynomial q(s) in Equation 6.1. 


Definition 6.3: 


Let G(s) be a real (p x m) proper rational matrix. 


« A right MFD P(s)Q~\(s) of G(s) is minimal if, and only if, the degree of det Q(s) in any other right 
MFD P(s)Q7) (s) of G(s) is at least as large as the degree of det Q(s). 

« A left MFD Qr'(s)P1(s) of G(s) is minimal if, and only if, the degree of det Qz(s) in any other left 
MFD OF (s)P (s) of G(s) is at least as large as the degree of det Qr(s). 


To formulate the matrix version of irreducibility, we require the following analogues of the coprimeness 
condition on the polynomials p(s) and q(s) appearing in Equation 6.1. 
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Definition 6.4: 


¢ Two polynomial matrices P(s) and Q(s) possessing m columns are said to be right coprime if, and only 
if, the following condition holds: If F(s) is an (m x m) polynomial matrix so that for some polynomial 
matrices P(s) and Q(s), P(s) = P(s)F(s) and Q(s) = Q(s)F(s), then F(s) is unimodular. 

¢ Two polynomial matrices P,(s) and Qr(s) possessing p rows are said to be left coprime if, and only 
if, the following condition holds: If F(s) is a (p x p) polynomial matrix so that for some polynomial 
matrices Pr(s) and Qx(s) P(s)= F(s)Pr(s) and Qr(s) = F(s)Qr(s), then F(s) is unimodular. 


Saying that two polynomial matrices are right (or left) coprime is the same as saying that they have no 
right (or left) common square polynomial matrix factors other than unimodular matrices. In this way, 
unimodular matrices play a role similar to that of nonzero real numbers in the definition of coprimeness 
for scalar polynomials. Any two unimodular matrices of the same size are trivially right and left coprime. 
Similarly, any (r x r) unimodular matrix is right coprime with any polynomial matrix that has r columns 
and left coprime with any polynomial matrix that has r rows. Equipped with Definition 6.4, we can explain 
what it means for a right or left MFD to be irreducible. 


Definition 6.5: 


Let G(s) be a real (p x m) proper rational matrix. 


« A right MFD P(s)Q71(s) of G(s) is irreducible if, and only if, P(s) and Q(s) are right coprime. 
« A left MFD Q,\(s)Px(s) of G(s) is irreducible if, and only if, P(s) and Qr(s) are left coprime. 


It is time to begin reconciling Definition 6.3, Definition 6.5, and the scalar intuition surrounding irre- 
ducibility and minimality of the fractional representation Equation 6.1. The proof of the following central 
result will occupy most of the remainder of this section; the finishing touches appear after Lemmas 6.1 
and 6.2 below. 


Theorem 6.1: 


Let G(s) be a real (p x m) proper rational matrix. 


« Aright MFD P(s)Q-1(s) of G(s) is minimal if, and only if, it is irreducible. Furthermore, if P(s)Q7! (s) 
and P(s)Q7! (s) are two minimal right MFDs of G(s), then an (m x m) unimodular matrix V(s) exists 
so that P(s) = P(s)V(s) and Q(s) = Q(s)V(s). 

« Aleft MFD Qe (s)PL (s) of G(s) is minimal if, and only if, it is irreducible. Furthermore, ifQ,! (s)Pr(s) 
and Qr!(s)Pr(s) are two minimal left MFDs of G(s), then a (p x p) unimodular matrix V(s) exists 
so that P,(s) = V(s)P(s) and Qr(s) = V(s)Q(s). 


Some parts of Theorem 6.1 are easier to prove than others. In particular, it is straightforward to show 
that a minimal right or left MFD is irreducible. To see this, suppose G(s) = P(s)Q7!(s) is a right MFD 
that is not irreducible. Since P(s) and Q(s) are not right coprime, factorizations P(s) = P(s)F (s) and 
Q(s) = O(s)F (s) exist with F(s) not unimodular. F(s) and Q(s) are evidently nonsingular because Q(s) is. 
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Furthermore, 
G(s) = P(s)Q71(s) = P(s)F(s)F~'(s)Q7|(s) = P(s)Q7'(s), 


which shows how to “reduce” the original MFD P(s)Q™!(s) by “canceling” the right common polynomial 
matrix factor F(s) from P(s) and Q(s). Because F(s) is not unimodular, det F(s) is a polynomial of positive 
degree, and the degree of det Q(s) is therefore strictly less than the degree of det Q(s). Consequently, the 
original right MFD P(s)Q~1(s) is not minimal. The argument demonstrating that minimality implies 
irreducibility for left MFDs is essentially identical. 

Proving the converse parts of the assertions in Theorem 6.1 is substantially more difficult. The approach 
we adopt, based on the so-called Smith form for polynomial matrices, is by no means the only one. 
Rugh [10] follows a quite different route, as does Vidyasagar [13]. We have elected to center our discussion 
on the Smith form partly because the manipulations we will be performing are similar to the things one 
does to obtain LDU decompositions [12] for matrices of numbers via Gauss elimination. 

We will be employing a famous result from algebra known as the Euclidean algorithm. It says that if 
f(s) isa real polynomial and g(s) is another real polynomial of lower degree, then unique real polynomials 
k(s) and p(s) exist, where p(s) has lower degree than g(s), for which 


f(s) = g(s)K(s) + p(s). 


It is customary to regard the zero polynomial as having degree —oo and nonzero constant polynomials 
as having degree zero. If g(s) divides f(s), then p(s) = 0; otherwise, one can interpret k(s) and p(s) 
respectively as the quotient and remainder obtained from dividing f(s) by g(s). 

The Smith form ofa real (k x r) polynomial matrix F(s) is a matrix obtained by performing elementary 
row and column operations on F(s). The row and column operations are of three kinds: 


¢ Interchanging the ith and jth rows (or the ith and jth columns). 

¢ Replacing row i with the difference (row i) —K(s)x (row j), where k(s) is a polynomial (or replacing 
column j with the difference (column j) —k(s)x (column i)). 

¢ Replacing row (or column) i with its multiple by a nonzero real number y. 


One can view each of these operations as the result of multiplying F(s) on the left or the right by a 
unimodular matrix. To interchange the ith and jth rows of F(s), multiply on the left by the permutation 
matrix I; 114 is a (k x k) identity matrix with the ith and jth rows interchanged. To replace row i with 
itself minus k(s) times row j, multiply on the left by the matrix E%[—K(s)], a (k x k) identity matrix except 
with —k(s) in the ij position. Because det 14 = —1 and det E4[—k(s)] = 1, both matrices are unimodular. 
To multiply row i by y, multiply F(s) on the left by the (k x k) diagonal matrix with y in the ith diagonal 
position and ones in all the others; this matrix is also unimodular, because its determinant is y. The 
column operations listed above result from multiplying on the right by (r x r) unimodular matrices of 
the types just described. 

The following result defines the Smith form ofa polynomial matrix. It is a special case of a more general 
theorem [6, pp. 175-180]. 


Theorem 6.2: Smith Form 


Let F(s) be a real (k x r) polynomial matrix with k > r. A (k x k) unimodular matrix U(s) and an (r x r) 
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unimodular matrix R(s) exist so that U(s)F(s)R(s) takes the form 


di(s) 0 Gdn 4 
OF a” “D> “seg? 6 


Oo 


0 0 dss) 0 0 
0 
A)=] 4 nee) (6.5) 
0 0 
0 0 


Furthermore, each nonzero dj(s) is monic, and d;(s) divides dj4,(s) for allj, 1<j<r—1. 


Proof 6.1. Let 38min be the smallest of the degrees of the nonzero polynomials in F(s). Now invoke the 
following procedure: 


Reduction Step 


Pick an entry in F(s) whose degree is min and use row and column exchanges to bring it to the (1, 1) 
position; denote the resulting matrix by G(s). For each i, 2 <i<k, use the Euclidean algorithm to find 
k;(s) so that pj(s) = [G(s)]ia — ki(s)[G(s)]11 has strictly lower degree than [G(s)]11, which has degree 8 nin. 
Observe that p;(s) = 0 if [G(s)]11 divides [G(s)]i1. Multiply G(s) on the left by the sequence of matrices 
E'![—x;(s)]; this has the effect of replacing each [G(s)]j1 with p;(s). Multiplying on the right by a similar 
sequence of E i_matrices replaces each [G(s)]1; with a polynomial whose degree is lower than 8 nin. 

The net result is a new matrix whose 8 min is lower than the 8pin of F(s). Repeating the reduction step 
on this new matrix results in a matrix whose 8min is still lower. Because we cannot continue reducing 83min 
forever, iterating the reduction step on the successor matrices of F(s) leads eventually to a matrix of the 
form 


d\(s) 0 O 0 
0 m™ © oi 
0 ™ Tw... T 
F\(s) = : (6.6) 
0 Tt Tt 


where each of the mts is a polynomial. 


Divisibility Check 


If d,(s) does not divide all the ms in Equation 6.6, find a m that d,(s) does not divide and multiply F;(s) 
on the left by an E'!-matrix so as to add the row containing the offending x to the first row of F1(s). Now 
repeat the reduction step on this new matrix. What results is a matrix of the form Equation 6.6 with a 
lower 8 min. 

Repeat the divisibility check on this new matrix. The process terminates eventually in a matrix of the 
form Equation 6.6 whose d)(s) divides all the polynomials 1. Note that this newest F)(s) can be written in 
the form U;(s)F(s)Ri(s) for some unimodular matrices Uj (s) and R,(s). The next phase of the Smith form 
computation entails performing the reduction step and divisibility check on the (k — 1 x r — 1) matrix of 
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ms in F)(s). What results is a matrix 


d,(s) 0 0 

0 dx(s) 0 0 

0 0 Tv Tt 

F,(s) = 0 0 Tt Tt 
0 0 Tt Tt 


in which d(s) divides all of the ms. Moreover, d,(s) divides d2(s) because d2(s) is a polynomial linear 
combination of the ms in Fj(s). Furthermore, there are unimodular matrices U2(s) and R2(s) so that 
F,(s) = U2(s)Fi(s)Ro(s), whereby F2(s) = U2(s) U1 (s)F(s)Ri (s)R2(s). 

Continuing in this fashion leads successively to F3(s),..., and finally F,(s), which has the same form 
as A(s). To get A(s), modify F,(s) by scaling all of the rows of F,(s) so that the nonzero d;(s)-polynomials 
are monic. It is evident that there are unimodular matrices U(s) and R(s) so that A(s) = U(s)F(s)R(s). 

A few comments are in order. First, if F(s) is a real (k x r) polynomial matrix with k < r, then applying 
Theorem 6.2 to F!(s) and transposing the result yields unimodular matrices U(s) and R(s) so that 
U(s)F(s)R(s) takes the form 


dy(s) 0 Os +. @. 0 0 
0 ds) 0 0 
awj=| ° 0 dls) 0. 6. . OF, (67) 
be we Se Be OO 
0 .  . 0 dls) 0. 0 


this is the Smith form of an F(s) with more columns than rows. Next, consider the polynomials {d;(s)} 
in Equation 6.5; these are called the elementary divisors of F(s). It might not appear prima facie as if 
the proof of Theorem 6.2 specified them uniquely, but it does. In fact, we have the following explicit 
characterization. 


Fact 6.1: 


For each j,1 <j < min(k,r), the product d,(s)...dj(s) is the monic greatest common divisor of all of the 
(j x j) minors in F(s). 


Proof 6.2. The divisibility conditions on the {dj(s)} make it obvious that d\(s)---dj(s) is the monic 
greatest common divisor of the (j x j) minors in A(s). Fact 6.1 follows immediately from the observation 
that the reduction procedure leading from F(s) to A(s) is such that for each i, the set of (j x j) minors of 
F;(s) has the same family of common divisors as the set of (j x j) minors of Fj+1(s). (See also the proof 
of Corollary 6.1 below.) As a result, the monic greatest common divisor of the (j x j) minors does not 
change over the course of the procedure. 


One final comment: the sequence of elementary divisors, in general, might start out with one or more 
1s and terminate with one or more Os. In fact, because every nonzero elementary divisor is a monic 
polynomial, each constant elementary divisor is either a 1 or a 0. If F(s) is a square matrix, then det F(s) 
is a nonzero real multiple of the product of its elementary divisors; this follows from the unimodularity 
of U(s) and R(s) in Theorem 6.2. Hence if F(s) is nonsingular, none of its elementary divisors is zero. 
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If F(s) is unimodular, then all of its elementary divisors are equal to 1. In other words, the Smith form of 
a (k x k) unimodular matrix is simply the (k x k) identity matrix I. 

The Smith form is the key to finishing the proof of Theorem 6.1. The following lemma, which offers 
useful alternative characterizations of coprimeness, is the first step in that direction. 


Lemma 6.1: 


Let P(s) and Q(s) be real polynomial matrices having respective sizes (p x m) and (m x m). The following 
three conditions are equivalent: 


1. P(s) and Q(s) are right coprime. 
2. There exist real polynomial matrices X(s) and Y(s) with respective sizes (m x m) and (m x p) so that 


X(s)Q(s) + Y(s)P(s) = Im. 
3. The (m+ p x m) matrix 
QUo) 
P(so) 
has full rank m for every complex number sy. 
Proof 6.3. We show that (1) ==> (2) => (3) => (1). Set 


F(s) = ro 


P(s) 


and, using the notation of Theorem 6.2, let A(s) = U(s)F(s)R(s) be the Smith form of F(s). Denote by A(s) 
the (m x m) matrix comprising the first m rows of A(s); the diagonal elements of A(s) are the elementary 
divisors of F(s). 

If (1) holds, we need A(s) = I. To see this, partition U—!(s) as 


> 


—1 _ “4 W\(s) W2/(s) 
0 @=we=| io He 


then 


F(s) = EA 


W\(s) 
P(s) 


-1 
hn A(s)R~(s), 


so that P(s) and Q(s) have A(s)R7!(s) is an (m x m) asa right common polynomial matrix factor. For (1) 
to hold, this last matrix must be unimodular, and that happens when A(s) = In. 

Hence, right coprimeness of P(s) and Q(s) implies that A(s) = I,,. It follows immediately in this case 
that if we partition the first m rows of U(s) as [Ui(s) U2(s)], then (2) holds with X(s) = R(s)U;(s) and 
Y(s) = R(s)U2(s). (3) is a straightforward consequence of (2) because, if F(s)) were rank-deficient for 
some So, then X(s9)Q(s9) + Y(so)P(so) could not be I,,. Finally, (3) implies (1) because, if (3) holds and 


_ [PG] 5 
F(s)= Ba R(s) 


is a factorization with R(s) (mx m) but not unimodular, then det R(s,) = 0 for at least one complex 
number sq, which contradicts (3). 


Not surprisingly, Lemma 6.1 has the following left-handed analogue. 
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Lemma 6.2: 


Let Pr(s) and Qr(s) be real polynomial matrices with respective sizes (p x m) and (p x p). The following 
three conditions are equivalent: 


1. Pry(s) and Qr(s) are left coprime. 

2. Real polynomial matrices X(s) and Y(s) exist with respective sizes (p x p) and (mx p) so that 
Qzu(s)X(s) + Pr(s)¥(s) = Ip. 

3. The (p x p+m) matrix 


[ Qr (So) Pr (so) | 


has full rank p for every complex number so. 


Weare ready now to finish proving Theorem 6.1. 


Proof of Theorem 6.1: We prove only the assertions about right MFDs. We have shown already that a 
minimal right MFD of G(s) is irreducible. As for the converse, suppose G(s) = P(s)Q7! is an irreducible 
right MFD and that G(s) = P(s)Q7} (s) is a minimal (hence irreducible) right MFD. By Lemma 6.1, there 
exist X(s), Y(s), X(s), and Y(s) of appropriate sizes satisfying 


X(s)Q(s) + Y¥(s)P(s) = Im 
X(s)Q(s) + ¥(s)P(s) = In- 


Invoke the fact that P(s)Q7!(s) = P(s)Q7! (s); a straightforward manipulation yields 


X(s)Q(s) + Y(s)P(s) = Qs) Q(s), 
X(s)Q(s) + ¥(s)P(s) = Q7(s)Q(s). 


The matrices on the right-hand sides are inverses of each other and are polynomial matrices; hence 
they must be unimodular. This implies, in particular, that the degrees of the determinants of Q(s) and 
Q(s) are the same, and P(s)Q7! (s) is, therefore, also a minimal right MFD. 

Finally, setting V(s) = Q71(s)Q(s) reveals that P(s) = P(s)V(s) and Q(s) = Q(s) V(s), proving that the 
two minimal right MFDs are related as in the theorem statement. 

Theorem 6.1 establishes the equivalence of minimality and irreducibility for MFDs. It is worth remark- 
ing that its proof furnishes a means for reducing a nonminimal MFD to a minimal one. The argument 
supporting Lemma 6.1 provides the key. Let G(s) = P(s)Q~!(s) be a nonminimal right MFD. Employing 
the same notation as in the proof of Lemma 6.1, 


def « 


x A(s)R71(s) = F(s)A(s)R71(s). 


Pet Ea is i (5) 


P(s) W3(s) 


Setting Q(s) = W\(s) and P(s) = W2(s) makes P(s)Q7! (s) aright MFD of G(s). By definition of W(s) and 
U(s), 


U(s)F(s) = re 


taken in conjunction with Theorem 6.2, this implies that F(s) has ones as its elementary divisors. As in 
the proof of Lemma 6.1, we conclude that P(s)Q71(s) is an irreducible (hence minimal) right MFD. 
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Theorem 6.1 also makes an important assertion about the “denominator matrices” appearing in min- 
imal MFDs. The theorem states that if Q(s) and Q(s) are denominator matrices in two minimal right 
MEDs of a proper rational G(s), then Q(s) and Q(s) differ by a right unimodular matrix factor. It follows 
that the determinants of Q(s) and Q(s) are identical up to a nonzero real multiple; in particular, det Q(s) 
and det Q(s) have the same roots including multiplicities. The same is true about the determinants of the 
denominator matrices Q;(s) and Q .(s) from two minimal left MFDs. It is clear that the poles of G(s) must 
lie among the roots of det Q(s) and det Qr(s). What is not so obvious is the fact, whose proof we postpone 
until the next section, that these last two polynomials are actually nonzero real multiples of each other. 


6.3 Fractional Degree and MacMillan Degree 


Conspicuously absent from Section 6.2 is any substantive discussion of relationships between right and 
left MFDs for a proper rational matrix G(s). The most important connection enables us to close the circle 
of ideas encompassing the minimal MFDs of Section 6.2 and the irreducible fractional representation 
Equation 6.1 for a scalar rational function. A crucial feature of Equation 6.1 is that the degree of q(s) is the 
same as the MacMillan degree of g(s). Our principal goal in what follows is to prove a similar assertion 
about the MacMillan degree of G(s) and the degrees of the determinants of the denominator matrices 
appearing in minimal right and left MFDs of G(s). 

Our first task is to demonstrate that, when P(s)Q7!(s) and Q'(s)Px(s) are right and left MFDs of a 
proper rational matrix G(s), the degrees of the polynomials det Q(s) and det Qz(s) are the same. To that 
end, we need the following technical lemma. 


Lemma 6.3: 


Suppose that the (m+ p x m+ p) polynomial matrix W(s) is nonsingular and that the (m x m) submatrix 
W\(s) is also nonsingular, where 


W(s)= bee WA 


W3(s) Wa(s) 
Then, 


1. H(s) = W4(s) — W3(s)W; 1(s) W2(s) is also nonsingular 
2. det W(s) = det W;(s) det H(s) 
3. W1(s) is given by 


W, |(s) + Wy '(s)W2(s)H~1(s)W3(s) Wy 1(s) —W, '(s)Wo(s)H~'(s) 
—H— (s)W3(s)W, (s) H7\(s) 


Proof 6.4. Statements (1) and (2) follow from the identity 


Im 0 _ | Wis) Was) 
ae a wo)=[ 0 Ho | 


because the matrix multiplying W(s) has determinant 1. Multiplying the last equation on the left by 


Ww, '(s) 0 Im —W2(s)H~'(s) 
0 H7\(s) 0 Ip 


yields statement (3). 


6-12 Control System Advanced Methods 


A direct consequence of Lemma 6.3 is the advertised relationship between determinants of the denom- 
inator matrices in right and left MFDs. 


Theorem 6.3: 


Let P(s)Q7!(s) and Qe '(s)Px(s), respectively, be minimal right and left MFDs of a real (p x m) proper 
rational matrix G(s). Then det Qr(s) is a nonzero real multiple of det Q(s); in particular, the two polynomials 
have the same degree and the same roots. 


Proof 6.5. We learned in the proof of Lemma 6.1 that unimodular matrices U(s) and R(s) exist so that 
Q(s) _ [In 
U(s) EA R(s) = | | : 


R(s) 0 
0 Ip 


Set 


M(s)= | U(s) 


and define W(s) = M~'(s); observe that M(s) and W(s) are unimodular. Partition M(s) and W(s) con- 
formably as follows: 


> 


Tene ae To) 


_ | Wis) Wals) 
M3(s) Ma(s) wo=| | 


W3(s) Wa(s) 


It follows that Wj (s) = Q(s) and W3(s) = P(s). In particular, W;(s) is nonsingular. By Lemma 6.3, M4(s) is 
nonsingular and equals H7—!(s), where H(s) = Wa(s) — W3 (s)W, *(s)Wo(s). Because M(s)W(s) = Im+p, 


M3(s)W2(s) + Ma(s)Wa(s) = Ips 
hence M3(s) and Ma(s) are left coprime by Lemma 6.1. Furthermore, 
M3(s) Wi(s) + Ma(s) W3(s) = 05 
the fact that G(s) = P(s)Q7!(s) = W3 (s)W; ‘(s) makes 
G(s) = My "(s)[-M3(s)] 


a minimal left MFD of G(s). By Theorem 6.1, det M,(s) is a nonzero real multiple of the determinant of 
the matrix Q;(s) appearing in any minimal left MFD Gr (s)Pzr(s) of G(s). 
By item (2) in Lemma 6.3, det W(s) = det W1(s) det H(s). Because Wj (s) = Q(s) and Mg(s) = H7\(s), 


det Q(s) = det W(s) det Mg(s). 


Unimodularity of W(s) implies that det Q(s) and det Mq(s) differ by a nonzero real multiple. 


Theorem 6.3 makes possible the following definition. 


Definition 6.6: 


The fractional degree §[G(s)] of a real (p x m) proper rational matrix G(s) is the degree of det Q(s) (or of 
det Qr(s)), where Q(s) (or Qr(s)) comes from a minimal right (or left) MFD P(s)Q7(s) (or Qr(s)Px(s)) 
of G(s). 
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As promised, we will demonstrate below that the fractional degree of a proper rational matrix G(s) is 
the same as its MacMillan degree 8y[G(s)]. Our approach will be first to show that 8p < $y and then to 
construct a state space realization for G(s) with state vector dimension 8-[G(s)]. The existence of such a 
realization guarantees that 8 > 5, from which it follows that the two degrees are equal. 

Accordingly, let G(s) be a given real (p x m) proper rational matrix. Let D = lims_,o9 G(s); set Z(s) = 
G(s) — D. If P(s)Q71(s) is a minimal right MED of G(s), then P(s)Q7! (s) isa minimal right MFD for Z(s), 
where P(s) = P(s) — DQ(s). To see why P(s)Qu} (s) is minimal, note that the two matrices 


Q(so) Q(So) 


F(s9) = bel » (so) = ies — DQ(s0) 


have the same nullspace (and hence the same rank) for every complex number so. It follows from item 
3 in Lemma 6.1 that P(s)Q7!(s) is irreducible and hence minimal. In addition, we can conclude that the 
fractional degrees of G(s) and Z(s) are the same. 

Suppose that n = 8y4(G(s)) and that (A, B, C, D) is a minimal realization of G(s). The Popov-Belevitch- 
Hautus test for reachability [11] implies that the (n x n+ m) matrix, 


K(so) = [ (soln _ A) B | 


has full rank n for every complex number sy. By Lemma 6.1, (sIy — A)~ 1 Bis an irreducible (hence minimal) 
left MFD of K(s). Thus K(s) has fractional degree n. Now, 


Z(s) = C(sIn — A) 'B = CF(s); 


it follows that the fractional degree of Z(s) is at most equal to n. To see this, suppose K(s) = P(s)Q7!(s) is 
a minimal right MFD of K(s); then Z(s) = [CP(s)]Q7!(s) isa right MFD of Z(s), and this MFD need not 
be minimal. Hence 8f(Z) < 5p(K) = n. The upshot is that the fractional degree of G(s), which is the same 
as the fractional degree of Z(s), is bounded from above by the MacMillan degree of G(s). In other words, 


Sp[G(s)] < 8u[G(s)]. 


Proving the reverse inequality requires a bit more effort. Suppose we have a minimal right MFD 
G(s) = P(s)Q7!(s) and corresponding right MFD Z(s) = P(s)Q71(s). The fractional degree of G(s) is the 
same as the degree of det Q(s). To show how the entries in Q(s) combine to determine the degree of 
det Q(s), we need the following definition. 


Definition 6.7: 


Let Q(s) be a real nonsingular (m x m) polynomial matrix. The jth column degree of Q(s), 8;[Q(s)], is the 
highest of the degrees of the polynomials in the jth column of Q(s), 1 <j < m. The high-order coefficient 
matrix of Q(s), Qu, is the real (m x m) matrix whose (i,j) entry is the coefficient of Sil) in [Q(s)]ij- 


The nonsingularity of Q(s) guarantees that all of the column degrees are nonnegative integers. The 
classic expansion for the determinant [12, page 157] reveals det Q(s) as the sum of m-fold products each 
of which contains precisely one element from each column of Q(s). It follows that the degree of det Q(s) 
is bounded from above by the sum of all the column degrees of Q(s). In fact, if 81, ... , 8, are the column 
degrees of Q(s), then the coefficient of the sit-+8m) in the expansion for det Q(s) is exactly det Quy, the 
determinant of the high-order coefficient matrix. In other words, det Q(s) has degree equal to the sum of 
the column degrees when det Qy 4 0, which is the same as saying that the high-order coefficient matrix 
is invertible. 
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Our method for constructing for G(s) a realization whose state dimension is §-[G(s)] hinges crucially 
on having a minimal right MFD P(s)Q7!(s) of G(s) whose Q(s) has an invertible high-order coefficient 
matrix. It turns out that such an MED always exists. The idea is to start with an arbitrary minimal right 
MFD P(s)Q71(s) and “operate on it” with a unimodular V(s) via 


P(s) = P(s)V(s), Q(s) = Q(s)V(s) 


to get another minimal right MFD P(s)Q7!(s) with an invertible Qy. We construct V(s) by looking at 
Q(s) only. Specifically, we prove the following assertion. 


Lemma 6.4: 


If Q(s) is a real nonsingular (m x m) polynomial matrix, a unimodular matrix V(s) exists so that 


© Qs)= Q(s)V(s) has an invertible high-order coefficient matrix Qu 
¢ The column degrees {8;(Q(s))} are in decreasing order, i.e., 


81(Q(s)) = 82(Q(s)) +++ 2 8m(QUs)). (6.8) 


Proof 6.6. If On is already invertible, let V(s) be a permutation matrix IT so that the columns of Q(s)T 
are lined up in decreasing order of column degree. If Qy is not invertible, after finding I as above, choose 
a nonzero w € R™ satisfying Qy Iw = 0. Assume without loss of generality that the first nonzero element 
in w is a 1 and occurs in the kth position. Let E(s) be the (m x m) polynomial matrix all of whose columns 
except the Ath are the same as those in the (m x m) identity matrix I,,; as for the the kth column, let 
[E(s)]jz be 0 when i < k and w;s**—* when i > k, where 8; denotes the jth column degree of Q(s)T1. E(s) 
has determinant 1 and is therefore unimodular; furthermore, Q(s)TIE(s) has the same columns as Q(s)IT 
except for the kth column, and the choice of E(s) guarantees that the kth column degree of Q(s)ME(s) is 
lower than the kth column degree of Q(s)1. 

The preceding paragraph describes a technique for taking a Q(s) with singular Qy and finding a 
unimodular matrix ITE(s) so that Q(s)TIE(s) has a set of column degrees whose sum is less than the sum 
of the column degrees of Q(s). If Q(s) = Q(s)TIE(s) still fails to have an invertible high-order coefficient 
matrix, we can repeat the column-permutation-and-reduction procedure on Q(s)ME(s), and so on. This 
iteration must terminate after a finite number of steps because we cannot reduce the sum of the column 
degrees forever. When all is said and done, we will have a unimodular matrix V(s) so that Q(s) = Q(s)V(s) 
has an invertible high-order coefficient matrix Qy with columns arrayed in decreasing order of column 
degree, so that the column degrees of Q(s) satisfy Equation 6.8. 


Thus we can take an arbitrary minimal right MFD G(s) = P(s)Q7! (s) and form a new minimal right 
MED P(s)Q™1(s) using P(s) = P(s)V(s) and Q(s) = Q(s)V(s); choosing V(s) appropriately, using Lemma 
6.4, makes Qy invertible, ensuring in turn that 


Sp[G(s)] = degree[det Q(s)] = 3:[Q(s)] ++ -- + 8m[Q(s)]. 


Furthermore, we can assume that the column degrees of Q(s) satisfy the ordering of Equation 6.8. 

Our final step is to produce a realization (A, B, C, D) of G(s) where A is (8¢ x 8p). This will confirm 
that 8-(G(s)) > 8:(G(s)) and, hence, that the two degrees are equal. First define D and Z(s) = G(s) — D 
as before, and let Z(s) = P(s)Q7}(s) be the corresponding minimal right MFD for Z(s). Because 


[P(s)]j = DIZ ixlQ() ly, 


k=1 


and because Z(s) is strictly proper, the degree of [P(s)]i is strictly less than 8; for all i and j. In particular, 
if 8; = 0, then [P(s)lij = 0 for all i, 1 <i < p (as usual, the zero polynomial has degree —oo). 
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Next define, for each k > 0, the polynomial k-vector s, by 


We now form a matrix S that has m columns and number of rows equal to 8, which is equal in turn to 
the sum of the column degrees {8;}. The jth column of $ has 8) + --- + 8;-1 0s at the top, the vector s5, 
in the next 3; positions, and 0s in the remaining positions. For example, if m = 3 and 8) = 3, 8) = 2, and 
$3 = 0, then 


1 0 O 
s 0 0 
S=|s? 0 0 
0 1 0 
0 s O 


Our observation above concerning the degrees of the entries in P(s) ensures that a real (p x 5p) matrix 
C exists so that P(s) = CS. This C will be the C-matrix in our realization of G(s). Since we want 


Z(s) = P(s)Q”'(s) = CSQ”'(s)B = C(sIs, — A) 'B, 


we will construct A and B so that 
SQ71(s) = (slsp — A) 'B, 
or, equivalently, 


sS = AS + BQ(s). (6.9) 


Recall first that Qy, the high-order coefficient matrix of Q(s), is invertible; by definition of Qy and the 
column degrees {8;}, 


si 0 0 
0 s® 0 
0 : y 
Q(s) = Qu ‘ + Q(s), (6.10) 
QO sdm-1 0 
0 bm 


where ((s) satisfies the same constraints as P(s) on the degrees of its entries. We may write Q(s) = GS for 
some real (m x 8) matrix G. Denote by X(s) the diagonal matrix on the right-hand side of Equation 6.10. 
Then A and B satisfy Equation 6.9 when 


sS = (A+ BQHG)S + BQH X(s). (6.11) 


Define B as follows: B= BOrs where B is the real (8p x m) matrix whose jth column is zero when 
3; = 0 and contains a single 1 at the 8; +---+ 8; position if 8; 4 0. Observe that if i= 8) +--- +; for 
some j, then the ith row of BQyG is the same as the jth row of G; for all other values of i, the ith row of 
BQuHG is zero. 
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Finally, define A. If i= 8; +---+ 8; for some j, then the ith row of A is the negative of the ith row of 
BQuG. For other values of i, the ith row of A contains a 1 in the (i, i+ 1) position (just above the diagonal) 
and 0s elsewhere. 

Verifying that A and B so defined satisfy the required relationship Equation 6.11 is straightforward. 
The important consequence of the construction is that Z(s) = C(sls, — A)~'B, so that (A, B,C, D) is a 
realization of G(s) whose A-matrix has size (8 x 5). The following theorem summarizes the results of 
this section so far. 


Theorem 6.4: 


If G(s) is a real (p x m) proper rational matrix, then the MacMillan degree of G(s) is the same as the 
fractional degree of G(s). The procedure outlined above yields a minimal realization (A, B, C, D) of G(s). 


The central role of Lemma 6.4 in the proof of Theorem 6.4 merits a closer look. In essence, Lemma 6.4 
guarantees the existence of a minimal right MFD G(s) = P(s)Q71(s) wherein the column degrees of Q(s) 
sum to the degree of the determinant of Q(s). The crucial enabling feature of Q(s) is the invertibility of its 
high-order coefficient matrix Qy. We will see presently that any minimal right MFD whose denominator 
matrix possesses this last property will have the same column degrees as Q(s) up to a reordering. As a 
result, these special column degrees are a feature of certain right MFDs and of the transfer function matrix 
G(s) itself. 


Definition 6.8: 


The ordered column indices l,[Q(s)], ...,lm[Q(s)] of a real nonsingular (m x m) polynomial matrix Q(s) 
are the column degrees of Q(s) arranged in decreasing order. 


Theorem 6.5: 


Let Q(s) and Q(s) be real nonsingular (m x m) polynomial matrices appearing in minimal right MFDs 
P(s)Q7!(s) and P(s)Q71(s) of a real (p x m) proper rational matrix G(s). If the high-order coefficient 
matrices Qy and Qy are both invertible, then the ordered column indices of Q(s) and Q(s) are identical. 


Proof 6.7. Assume first that Q(s) and Q(s) satisfy the second item in Lemma 6.4, ie., have respective 
column degrees {8;} and {8;} that are decreasing in j. Write Equation 6.10 along with a similar equation 
for Q(s) as follows: 


Q(s) = Qu X(s) + As) 
Q(s) = Qu H(s) + Qs). 


By construction, 


lim Q(s)=7K(s) {lim A(s) =0, 

|s| oo |s| 00 

im Q(s)S-1s) & Jim NO=0. 
s|—> Co 


|s| oo 
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Because Q(s) and Q(s) both come from minimal right MFDs of G(s), by Theorem 6.1 an (m x m) 
unimodular matrix V(s) exists so that Q(s) = Q(s)V(s). Manipulation yields 


X(s)U(s) 271s) = En + A(S)17 QF Ou Um + A(s)]. (6.12) 


The right-hand side of Equation 6.12 approaches a constant limit as |s| > 00; note, in particular, that 
Im + A(s) is nonsingular for |s| large enough. Meanwhile, the (i,j) entry of the matrix on the left-hand 
side of Equation 6.12 is simply si-5[U(s)]y.- Hence we need [U(s)];; = 0 whenever 8; > 8;. One by one 
we will show that 8; < 8. If 8; > 8), then, by the ordering on the §s, 81 > 8; for all j, and the entire first 
row of U(s) must be zero, contradicting nonsingularity and, a fortiori, unimodularity of U(s). Assume 
inductively that 8; < §; for j < k but that 8% > §x. In this case, the orderings on the 8s and §s imply that 
8; > Be > 8; for all i < k and all j > k; hence the entire upper right-hand (k x m—k) corner of U(s) must 
be zero, which contradicts unimodularity of U(s) once again. 

Thus §; < 8; for all j, 1 <j < m. It follows that 8; = 8; for every j because the sum of the 8s and the sum 
of the 8s must both equal the fractional degree of G(s), which is the common degree of the determinants 
of Q(s) and Q(s). Our initial assumption that the columns of Q(s) and Q(s) were arrayed in decreasing 
order of column degree means, in terms of Definition 6.8, that Q(s) and Q(s) have the same ordered 
column indices. Finally, it is easy to eliminate this initial assumption; simply precede the argument with 
right multiplications by permutation matrices IT and I that reorder the matrices’ columns appropriately. 
In any event, the ordered column indices of Q(s) and Q(s) are identical. 


The principal consequence of Theorem 6.5 is that any two Q(s)-matrices with invertible Qys appearing 
in minimal right MFDs of G(s) have the same set of ordered column indices. These special ordered column 
indices are sometimes called the Kronecker controllability indices of G(s). They have other names, as well; 
Forney [4], for example, calls them invariant dynamical indices. They relate to controllability because 
the realization (A, B, C, D) we constructed en route to Theorem 6.4 is precisely the MIMO analogue to 
the SISO realization given in Equations 6.2 and 6.3. The realizations are called controllable canonical 
forms [2,7,10]. Interested readers can verify that applying the realization procedure following Lemma 6.4 
to a scalar irreducible fractional representation Equation 6.1 leads exactly to Equations 6.2 and 6.3. 

It is worth making one final observation. Our proof of Theorem 6.4 relied on constructing a minimal 
realization of a proper rational transfer matrix G(s) starting from a minimal right MFD P(s)Q71(s) of 
G(s). We could have worked instead with a minimal left MFD G(s) = Qr'(s)Pr(s)s in which case we 
would have considered the row degrees and high-order coefficient matrix of Q;(s). Perhaps the simplest 
way to view this is to realize G"(s) by following the route we have already laid out beginning with the 
right MED P} (s)(Q}(s))~! of G(s) and subsequently transposing the end result. 


6.4 Smith-MacMillan Form, ARMA Models, and Stable 
Coprime Factorization 


The aim of this section is to tie up some loose ends and to point the reader toward some important 
modern control theoretic developments that rest heavily on the theory of polynomial MFDs described in 
the foregoing sections. First we discuss the so-called Smith-MacMillan form for proper rational matrices; 
many of the results detailed in Sections 6.2 and 6.3 have alternative derivations based on the Smith- 
MacMillan form. Next, we describe briefly the connection between MFDs and ARMA models for MIMO 
linear systems. We close with a quick introduction to stable coprime factorization and mention briefly its 
generalizations and applications in robust control theory. 

The Smith-MacMillan form of a real (p x m) proper rational matrix G(s) is basically a rational ver- 
sion of the Smith form of a polynomial matrix F(s). It was introduced originally by MacMillan [9] and 
later exploited by Kalman in an important paper [8] that demonstrated correspondences between several 
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notions of rational matrix degree. Given G(s), begin by letting q(s) be the monic lowest common denom- 
inator of its entries. Set F(s) = q(s)G(s); by Theorem 6.2, we can find unimodular matrices U(s) and R(s) 
of respective sizes (p x p) and (m x m) so that A(s) = U(s)F(s)R(s) has the form of Equation 6.5 or 6.7 
depending on whether p > m or p < m, respectively. 


Assuming temporarily that p > m, the matrix U(s)G(s)R(s) =  F(s) = Asm(s) takes the form 


q(s) 
yils)/o1(s) 0 0 ane 0 
0 y2(s)/2(s) 0 ; 0 
0 0 y3(s)/3(s) 0 0 
, : ; rae 0 
0 1 OO pel fantal? (6.13) 
0 . ; i, 0 
0 _ 0 


where yx(s)/o,(s) is the fraction d;(s)/q(s) expressed in lowest terms. If dy(s) = 0, set b,(s) = 1. The 
divisibility conditions on {d;,(s)} guaranteed by Theorem 6.2 ensure that y;(s) divides yj41(s) and +1 (s) 
divides o;(s) for allk, l1<k<m. 

Furthermore, if we set 


yils) 0 0 fois 0 
0 y2(s) 0 use hg 0 
0 0 y3(s) O.. 0 
P(s) = U-\(s) : a 
0 . 0 
0 0 


def -1(s) (5) 


and 
1(s) 0 0 es 0 
0 (s) O .. 0 
gs=n} © 9 &G) 9 : 
, . : : : 0 
0 . . - 0 bm(s) 
© R(s)@(s), 


then P(s)Q7!(s) isa right MED of G(s). 
P(s)Q™1(s) is minimal because of the divisibility conditions on the {yj} and {0}. The easiest way to see 
this is by checking that the (m+ p x m) matrix 


Q(so)} _ | R(so) 0 D(so) 

Pls) | 0. UE) LTGod 
has full rank m for every complex number sy. The idea is that if, for example, y,(s.) = 0 for some smallest 
value of k, then y;(so) = 0 for all j = k; hence $;(so) 4 0 for j = k which, coupled with y;(sy) 4 0 forj < k, 
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means that m of the {y;(so)} and {$;(so)} are nonzero. Because P(s)Q7!(s) is a minimal right MFD of 
G(s), it follows from Theorem 6.4 that the MacMillan degree of G(s) is the degree of det Q(s), which is 
the sum of the degrees of the polynomials {¢;(s)}. 

The same sort of analysis works when p < m; in that case, Asy(s) = U(s)G(s)R(s) looks like a rational 
version of the matrix in Equation 6.7. In either case, Asqs(s) is called the Smith-MacMillan form of G(s). 
To summarize, 


Theorem 6.6: Smith-MacMillan Form 


Let G(s) be a real (p x m) proper rational matrix and let m A p be the minimum of m and p. Unimodular 
matrices U(s) and R(s) exist so that Asy(s) = U(s)G(s)R(s), where 
¢ [Asm(s)]ig = 0, except when i=j and [Asm(s)]j(s) = yi(s)/oj(s), 1S js mAp, where the 
{y;(s)/;(s)} are ratios of coprime monic polynomials, 
e yj(s) divides yj+1(s) and ;+1(s) divides o;(s), 1<j<map, 
¢ The MacMillan degree of G(s) is the sum of the degrees of the {;(s)}. 


An interesting consequence of Theorem 6.6 is the following characterization of MacMillan degree [9]. 


Corollary 6.1: 


The MacMillan degree of a real (p x m) proper rational matrix G(s) is the degree of the lowest common 
denominator of all of the minor subdeterminants of G(s). 


Proof 6.8. Assume p > m; the argument is similar when p < m. A glance at Asy(s) in Equation 6.13 and 
the divisibility conditions on the {o;(s)} in Theorem 6.6 reveals that for each k, 1 < k < m, the product 
1 (s)- ++ bx(s) is the monic lowest common denominator of all of the (j x j) minors in Asy(s) of order 
j < k. Hence the product of all of the ¢;(s) is the monic lowest common denominator of all of the minors 
of Asm(s). Now, G(s) and Asa(s) are related via pre- and postmultiplication by unimodular matrices; 
hence the (j x j) minor determinants of each matrix are polynomial linear combinations of the (j x /) 
minors of the other matrix. It follows that any common denominator for one set of minors is a common 
denominator for the other set, and the proof is complete. 


Observe that we could have generated many of the results in Sections 6.2 and 6.4 by appealing to the 
Smith-MacMillan form. A disadvantage of this approach is that Theorem 6.6 produces only right MFDs 
for (p x m) rational matrices when p > m and only left MFDs when p < m. 

We consider next some relationships between MFDs and some well-known time-domain representa- 
tions of input-output linear systems. Let G(s) be a real (p x m) proper rational matrix that is the transfer 
function of a real m-input, p-output, time-invariant linear system with input u: R > R”™ and output 
y:R— R?. Let L{y}(s) and £{u}(s) be the Laplace transforms of y and u, so L{y}(s) = G(s)L{u}(s). If 
G(s) = Q, '(s)P1(s) is a left MED of G(s), then 


Qu(s)L{y}(s) = Pr(s)L{u}(s) ; 
in the time domain, this last equation corresponds with the vector differential equation 
Qi (D) y(t) = Pr(D) u(t), (6.14) 


where D is the differential operator £. Equation 6.14 is an ARMA (autoregressive moving-average) 
representation for the system’s input-output relationship. 
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Similarly, if G(s) = P(s)Q7!(s) is a right MFD of G(s), we can define w: R-> R™ by means of the 
autoregressive (AR) differential equation 


Q(D) w(t) = u(t) (6.15) 
and use w as the input to a moving-average (MA) specification of y, namely, 

y(t) = PD) w(t). (6.16) 
Because 


Liy}(s) = P(s)L{w}(s) and L{w}(s) = Q-'(s)L{uh(s), 
Liy}(s) = P(s)Q | (s)L{u}(s) = G(s)L{u}(s); 


so Equations 6.15 and 6.16 together constitute another time-domain description of the input-output 
behavior of the system. Whereas Equation 6.14 gives an ARMA description for the system, Equations 6.15 
and 6.16 split the input-output relation into autoregressive and moving-average parts. For a SISO system, 
any fractional representation of the form Equation 6.1 acts as a left and right “MFD,” so that the two 
time-domain characterizations are identical. 

We close by presenting a very brief introduction to some of the ideas underlying stable coprime 
factorization, which is the single most important off-shoot of the theory of MFDs for input-output 
systems. We call a rational matrix H(s) stable if, and only if, the poles of the entries in H(s) lie in the 
open left half-plane Re{s} < 0. As usual, let G(s) be a real (p x m) proper rational matrix. It turns out to 
be possible to write G(s) in the form G(s) = Hy (s)H;' (s), where 


e H,(s) and Hp(s) are stable proper rational matrices of respective sizes (p x m) and (m x m), 

¢ Hj(s) and Hp(s) are right coprime over the ring of stable rational functions, that is, the only common 
right stable (m x m) proper rational matrix factors of H)(s) and H2(s) have inverses that are also 
stable and proper. 


Any such representation G(s) = Hy (s)Hy \(s) is called a stable right coprime factorization of G(s). One 
can define stable left coprime factorizations similarly. 

It is not difficult to show that stable coprime factorizations exist. One approach, patterned after a 
technique due originally to Vidyasagar [14], goes as follows. Given G(s), choose a > 0 so that —a is not 
a pole of any entry in G(s). Let o=1/(s +4), so that s= (1 —ao)/o. Define G(o) = G((1 — ao)/o). It 
follows that G(o) is a proper rational matrix function of o. To see why it is proper, observe that the 
condition o — oo is the same as s > —a, and —a is not a pole of G(s). 

Now invoke the theory of Section 6.2 and find a minimal right MFD G(c) = P(o)Q71(c) of Go). 
Finally, set Hy(s) = P[1/(s-+a)] and H2(s) = Q[1/(s+a)]. Then 


G(s) =G (=) = H,(s)Hy\(s). 


Because P(o) and Q(o) are polynomial in o, H,(s) and H(s) are proper rational matrix functions of s. 
Moreover, all of the poles of H;(s) and Hj(s) are at —a, which means that H;(s) and Hp(s) are stable. 
Furthermore, any stable (m x m) proper rational right common factor H(s) of H(s) and H2(s) defines, 
via V(s) = H((1—0)/o), a polynomial right common factor of P(o) and Q(o), which must have a 
polynomial inverse by minimality of P(o)Q7!(o). It follows that H~1(s) = V—![1/(s + @)] is stable and 
proper, implying that H;(s) and Hp(s) are right coprime over the ring of stable proper rational functions. 

The principal application of stable coprime factorizations is to robust control system design. At the 
heart of such applications is the notion of the H® norm ofa stable proper rational matrix H(s). Given such 
an H(s), the H® norm of H(s) is the supremum over w € R of the largest singular value of H(iw). Given 
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two possibly unstable (p x m) transfer function matrices Gg(s) and G,(s), one can define the distance 
between G,(s) and G;(s) in terms of the H® norm of the stable rational matrix 


ea) _ [at 

Ha2(s)}_— LHpa(s)]’ 

where G,(s) = Hai(s)H3 (s) and G,(s) = Hy (s)H,," (s) are stable coprime factorizations of G(s) and 
Gy(s). 

Interested readers can consult [1,3,5,13], and the references therein for a through development of the 
ideas underlying robust control system design and their dependence on the theory of stable coprime 
factorization. A by-product of Vidyasagar’s approach [13] is a framework for understanding MFDs and 
stable coprime factorizations in terms of more general themes from abstract algebra, notably ring theory. 
This framework reveals that many of our results possess natural generalizations that apply in contexts 
broader than those considered here. 


6.5 Defining Terms 


Proper rational matrix: A matrix whose entries are proper rational functions, i.e., ratios of polynomials 
each of whose numerator degrees is less than or equal to its denominator degree. 

MacMillan degree: The dimension of the state in a minimal realization of a proper rational transfer 
function matrix. 

Real polynomial matrix: A matrix whose entries are polynomials with real coefficients. 

Nonsingular: A real square polynomial matrix is nonsingular if its determinant is a nonzero polynomial. 

Unimodular: A real square polynomial matrix is unimodular if its determinant is a nonzero real number. 
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7.1 Motivations and Preliminaries 


Over the last decades, stability and performance of control systems affected by bounded perturbations 
have been studied in depth. The attention of researchers and control engineers concentrated on robustness 
tools in the areas Hoo, Kharitonov (or real parametric uncertainty), L}, Lyapunov, j1, and quantitative 
feedback control (QFT). For further discussions on these topics and on the exposition of the main 
technical results, the reader may consult different sections of this volume and the special issue on robust 
control of Automatica (1993). 

One of the key features of this chapter is the concept of robustness. To explain, instead of a single 
(nominal) system, we study a family of systems and we say that a certain property (e.g., stability or perfor- 
mance) is robustly satisfied if it is satisfied for all members of the family. In particular, we focus on linear, 
time-invariant, single-input, single-output systems affected by real parametric uncertainty. Stability of 
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interval polynomials (i.e., polynomials whose coefficients lie within given intervals) and the well-known 
Theorem of Kharitonov (Kharitonov, 1978) are at the core of this research area. This theorem states that 
an interval polynomial has all its roots in the open left half-plane if and only if four specially constructed 
polynomials have roots in the open left half-plane. Subsequently, the Edge Theorem (Bartlett et al., 1988) 
studied the problem of affine dependence between coefficients and uncertain parameters, and more gen- 
eral regions than the open left half-plane. This result provides an elegant solution proving that it suffices to 
check stability of the so-called one-dimensional exposed edges. We refer to the books (Ackermann, 1993; 
Barmish, 1994; Bhattacharyya et al., 1995; Djaferis, 1995; Kogan, 1995) for a discussion of the extensive 
literature on this subject. 

To explain robustness analysis more precisely with real parametric uncertainty, we consider a family of 
polynomials p(s, q) of degree n whose real coefficients a;(q) are continuous functions of an ¢-dimensional 
vector of real uncertain parameters q, each bounded in the interval [q; , q;' ]. More formally, we define 


p(s, q) = ao(q) + a1(q)s + a(q)s? +++» +an(q)s", 
q a [q1; q2>--- qe); 


and the set 
OA \otg. <p eg iS 2 hh 


We assume that p(s, q) is of degree n for all q € Q—that is, we assume that a,(q) £0 for all qe Q. 
Whenever the relations between the polynomial coefficients a;(q) and the vector q are specified, we study 
the root location of p(s, q) for all q € Q. Within this framework, the basic property we need to guarantee 
is robust stability. In particular, we say that p(s, q) is robustly stable if p(s, q) has roots in the open left 
half-plane for all q € Q. 

The real parametric approach can be also formulated for control systems. In this case, we deal with a 
family of plants denoted by P(s, q). More precisely, we concentrate on robust stability or performance of 
a proper plant 
Np(s, g) 

Dp(s,q)’ 

where Np(s, q) and Dp(s, q) are the numerator and denominator polynomials whose real coefficients are 
continuous functions of g. We assume that Dp(s, q) has invariant degree for all q € Q. We also assume 
that there is no unstable pole-zero cancellation for all q € Q; the reader may refer to Chockalingam and 
Dasgupta (1993) for further discussions. 

Robustness analysis is clearly of interest when the plant requires compensation. In practice, if the 
compensator is designed on the basis of the nominal plant, then, robustness analysis can be performed 
by means of the closed-loop polynomial. That is, given a compensator transfer function 


P(s,q) = 


Nc(s) 
Dc(s) 


connected in a feedback loop with P(s, q), we immediately write the closed-loop polynomial 


C(s) = 


p(s, q) = Np(s, q)Nc(s) + Dp(s, q)Dc(s) 


whose root location determines closed-loop stability. 

To conclude this preliminary discussion, we remark that one of the main technical tools described here 
is the so-called value set (or template in the QFT jargon, see Horowitz, 1991; see Barmish, 1994 for a 
detailed exposition of its properties). In particular, we show that if the polynomial or plant coefficients are 
affine functions of q, the value set can be easily constructed with 2D graphics. Consequently, robustness 
tests in the frequency domain can be readily performed. Finally, in this chapter, since the main goal is to 
introduce the basic concepts and tools available for robustness analysis with real parametric uncertainty, 
we do not provide formal proofs but we make reference to the specific literature on the subject. 
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7.1.1 Motivating Example: DC Electric Motor with Uncertain Parameters 


For the sake of illustrative purposes, an example of a DC electric motor is formulated and carried 
out throughout this chapter in various forms. Consider the system represented in Figure 7.1 of an 
armature-controlled DC electric motor with independent excitation. The voltage to angle transfer function 
P(s) = ©(s)/V(s) is given by 


K 


PO) = TST (RPA BDS FREER’ 


where L is the armature inductance, R the armature resistance, K the motor electromotive force-speed 
constant, J the moment of inertia, and B the mechanical friction. Clearly, the values of some of these 
parameters may be uncertain. For example, the moment of inertia and the mechanical friction are func- 
tions of the load. Therefore, depending on the specific application, if the load is not fixed, the values of J 
and B are not precisely known. Similarly, the armature resistence R is a parameter that can be measured 
very accurately but is subject to temperature variations, and the motor constant K is a function of the 
field magnetic flow which may vary. 

To summarize, it is reasonable to say that the motor parameters, or a subset of them, may be unknown 
but bounded within given intervals. More precisely, we can identify 


g=l; q@=R g=K; ga=J; gs=B 


and specify a given interval [q; , q;' ] for each qi,i = 1,2,...,5. Then, instead of P(s), we write 


93 


P(s,q) = ; 5 ri : 
gigas? + (qoga + 9195)s* + (q5 + 9295)s 


7.2 Description of the Uncertainty Structures 


As discussed in the preliminaries in Section 7.1, we consider a proper plant P(s,q) whose coefficients 
are continuous functions of the uncertainty q which is confined to the set Q. Depending on the specific 
problem under consideration, the coefficients of Np(s, q) and Dp(s, q) may be linear or nonlinear functions 
of q. To explain more precisely, we consider the example of Section 7.1.1. Assume that the armature 


i= Armature current 
——> 


V = Armature voltage 
@ = Angular speed 


C,, = Motor torque 


0 = Angle 


C,.= Load wraue \\ 


FIGURE 7.1 DC electric motor. 
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inductance L, the armature resistance R, and the constant K are fixed while the moment of inertia J and 
the mechanical friction B are unknown. Then, we take q; =J and q2 = B as uncertain parameters; the 
resulting set Q is a two-dimensional rectangle. In this case, the plant coefficients are affine* functions of 
qi and qo 


Np(s, q) = K; 
Dp(s, q) = Las? + (Rai + Lqa)s” + (K? + Rqp)s 


and we say that the plant has an affine uncertainty structure. This situation arises in practice whenever, for 
example, the load conditions are not known. Other cases, however, may be quite different from the point 
of view of the uncertainty description. For example, if L, K, and B are fixed and R and J are uncertain, we 
identify q) and q with R and J, respectively. We observe that the denominator coefficient of s? contains 
the product of the two uncertain parameters q; and q2 


Np(s, q) = K; 
Dp(s, q) = Lqs? + (qig2+ BL)s* + (K? + Bqi)s. 


In this case, the plant coefficients are no longer affine functions of the uncertainties but they are multiaffine 
functions’ of q. This discussion can be further generalized. It is well known that the motor constant K 
is proportional to the magnetic flow. In an ideal machine with independent excitation, such a flow is 
constant, but in a real machine, the armature reaction phenomenon causes magnetic saturation with the 
consequence that the constant K drops when the armature current exceeds a certain value. Hence, we 
consider K as an uncertain parameter and we set q; = K. In turn, this implies that the plant coefficients are 
polynomial functions of the uncertainties. In addition, since q; enters in Np(s, q) and Dp(s, q), we observe 
that there is coupling between numerator and denominator coefficients. In different situations when 
this coupling is not present, we say that the numerator and denominator uncertainties are independent. 
An important class of independent uncertainties, in which all the coefficients of the numerator and 
denominator change independently within given intervals, is the so-called interval plant; for example, 
see Barmish et al. (1992). In other words, an interval plant is the ratio of two independent interval 
polynomials; recall that an interval polynomial 


p(s, q) = qo + qis+ qos? + +--+ qns" 


has independent coefficients bounded in given intervals q; < qi < q} fori=0,1,2,...,n. 

The choice of the uncertain parameters for a control system is a modeling problem, but robustness 
analysis is of increasing difficulty for more general uncertainty structures. In the following sections, we 
show that this analysis can be easily performed if the structure is affine and we demonstrate that a “tight” 
approximate solution can be readily computed in the multiaffine case. 


7.3 Uncertainty Structure Preservation with Feedback 


In the previous sections, we described the classes of uncertainty structures entering into the open-loop 
plant. From the control system point of view, an important and closely related question arises: what 
are the conditions under which a certain uncertainty structure is preserved with feedback? To answer 
this question, we consider a plant P(s, q) and a compensator C(s) connected with the feedback structure 
shown in Figure 7.2. 


* An affine function f : Q > R is the sum ofa linear function and a constant. For example, f(q) = 3q1 + 2q2 — 4 is affine. 


¥ A function f : Q—> R is said to be multiaffine if the following condition holds: If all components q,...,q except for 
one are fixed, then f is affine. For example, f (q) = 3914243 — 6q1q3 + 4q2q3 + 2q) — 2q2 + q3 — 1 is multiaffine. 
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FIGURE 7.2 Closed-loop system. 


Depending on the specific problem under consideration (e.g., disturbance attenuation or tracking), 
we study sensitivity, complementary sensitivity, and output-disturbance transfer functions (e.g., see 
Doyle et al., 1992) 


T(s,@)2 P(s, q)C(s) 


te P(s, q) 
~ T+ P(s5, Cs)’ Ris, 4) 


S(s,q) = ~ 1+P(s, Cl) 


1 
1+ P(s, q)C(s) ; 
For example, it is immediate to show that the sensitivity function S(s, q) takes the form 


Dp(s, q)Dc(s) 


SSD = New gels) + Dele @Dc@) 


If the uncertainty q enters affinely into the plant numerator and denominator, q also enters affinely 
into S(s,q). We conclude that the affine structure is preserved with feedback. The same fact also holds 
for T(s,q) and R(s,q). Next, we consider the interval plant structure; recall that an interval plant has 
independent coefficients bounded in given intervals. It is easy to see that, in general, this structure is not 
preserved with compensation. Moreover, if the plant is affected by uncertainty entering independently 
into numerator and denominator coefficients, this decoupling is destroyed for all transfer functions 
S(s, q), T(s, q), and R(s, q). Finally, it is important to note that the multiaffine and polynomial uncertainty 
structures are preserved with feedback. Table 7.1 summarizes this discussion. In each entry of the first row 
of the table we specify the structure of the uncertain plant P(s, q) and in the entry below the corresponding 
structure of S(s, q), T(s, q), and R(s, q). 


7.4 Overbounding with Affine Uncertainty: The Issue 
of Conservatism 


As briefly mentioned at the end of Section 7.3, the affine structure is very convenient for performing 
robustness analysis. However, in several real applications, the plant does not have this form; for example, 
see Abate et al. (1994). In such cases, the nonlinear uncertainty structure can always be embedded into 
an affine structure by replacing the original family by a “larger” one. Even though this process has the 
advantage that it handles much more general robustness problems, it has the obvious drawback that it gives 
only an approximate but guaranteed solution. Clearly, the goodness of the approximation depends on the 


TABLE7.1 Uncertainty Structure with Feedback 


P(s, q) Independent Interval Affine Multiaffine Polynomial 


S(s, q), T(s, q), R(s, q) Dependent Affine Affine Multiaffine Polynomial 
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specific problem under consideration. To illustrate this simple overbounding methodology, we consider 
the DC-electric motor transfer function with two uncertain parameters and take qj = R and qz = J. As 
previously discussed, with this specific choice, the plant has a multiaffine uncertainty structure. That is, 


K 


P(s,g) = : 
(sq) Laas? + (qiq2 + BL)s? + (K? + Bqi)s 


To overbound P(s, q) with an affine structure, we set q3 = qiq2. Given bounds [q; qi | and [q, , qs | for 
qi and qp, the range of variation [43 , q; ] for q3 can be easily computed: 
gs = ming, 9, DT HH GS 
g3 =maxl9, 1 > DT G- 
Clearly, the new uncertain plant 


K 
Lqzs? + (q3 + BL)s? + (K? + Bqi)s 


P(s,q) = 


has three uncertain parameters q = (q1, q2, q3) entering affinely into P(s,q). This new parameter is not 
independent, because q3 = q1q2 and not all values of [q; , q | are physically realizable. However, since we 
assume that the coefficients qj are independent, this technique leads to an affine overbounding of P(s, q) 
with P(s,q). We conclude that if a certain property is guaranteed for P(s,q), then, this same property is 
also guaranteed for P(s, q). Unfortunately, the converse is not true. The control systems interpretation 
of this fact is immediate: Suppose that a certain compensator C(s) does not stabilize P(s,q). It may turn 
out that this same compensator does stabilize the family P(s, q). Figure 7.3 illustrates the overbounding 
procedure for q, = 1.2, qt = 17,4, =1,7, qa = 2.2,q, = 2.04 and qa = 3.74. 

To generalize this discussion, we restate the overbounding problem as follows: Given a plant P(s, q) 
having nonlinear uncertainty structure and a set Q, find a new uncertain plant P(s, q) with affine uncer- 
tainty structure and a new set Q. In general, there is no unique solution and there is no systematic 
procedure to construct an “optimal” overbounding. In practice, however, the control engineer may find 
via heuristic considerations a reasonably good method to perform it. The most natural way to obtain 
this bound may be to compute an interval overbounding for each coefficient of the numerator and 
denominator coefficients—that is, an interval plant overbounding. To illustrate, letting aj(q) and b;(q) 


FIGURE 7.3. Overbounding procedure. 


Robustness Analysis with Real Parametric Uncertainty 7-7 


denote the numerator and denominator coefficients of P(s, q), lower and upper bounds are given by 
ee ee 
a, = ean aji(q); a; ae aj(q) 
and 
b> =minbj(q); bt = b;(q). 
; min i(q); 0; pe i(q) 
If a;(q) and b;(q) are affine or multiaffine functions and q lies in a rectangular set Q, these minimizations 


and maximizations can be easily performed. That is, if we denote by q!,q?,...,q' + q? the vertices of 
Q, then 


a; meen a;(q) ns alg’); a; pane aj(q) pe: ai(q’) 

and 

b> =minb(q)= min 5,(q‘); br = bi(q) = bi(q*). 
ee (q) Re (q')s 0; ens (q) ae (q") 


1 


To conclude this section, we remark that for more general uncertainty structures than multiaffine, a 
tight interval plant overbounding may be difficult to construct. 


7.5 Robustness Analysis for Affine Plants 


In this section, we study robustness analysis of a plant P(s, q) affected by affine uncertainty q € Q. The 
approach taken here is an extension of the classical Nyquist criterion and requires the notion of value set. 
For fixed frequency s = jw, we define the value set P(jw, Q) C C as 


P(ja, Q) = {P(ja, q) : Dp(jw, gq) £ 0,4 € Qh. 


Roughly speaking, P(jw, Q) is a set in the complex plane which graphically represents the uncertain 
plant. Without uncertainty, P(jw) is a singleton and its plot for a range of frequencies is the Nyquist 
diagram. The nice feature is that this set is two-dimensional even if the number of uncertain parameters 
is large. Besides the issue of the value set construction (which is relegated to the next subsection for the 
specific case of affine plants), we now formally state a robustness criterion. This is an extension of the 
classical Nyquist criterion and holds for more general uncertainty structures than affine—continuity of 
the plant coefficients with respect to the uncertain parameters suffices. However, for more general classes 
of plants than affine, the construction of the value set is a hard problem. 


Criterion 7.1: Robustness Criterion for Uncertain Plants 


The plant P(s, q) is robustly stable for all q € Q if and only if the Nyquist stability criterion is satisfied for 
some q € Qand —1+j0 ¢ P(jw, Q) forallw eR. 


This criterion can be proved using continuity arguments; see Fu (1990). To detect robustness, one 
should check if the Nyquist stability criterion holds for some q € Q; without loss of generality, this check 
can be performed for the nominal plant. Second, it should be verified that the value set does not go through 
the point —1 +0 for all w € R. In practice, however, one can discretize a bounded interval 2 C R witha 
“sufficiently” large number of samples—continuity considerations guarantee that the intersampling is not 
a critical issue. Finally, by drawing the value set, the gain and phase margins can be graphically evaluated; 
similarly, the resonance peak of the closed-loop system can be computed using the well-known constant 
M-circles. 
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7.5.1 Value Set Construction for Affine Plants 


In this section, we discuss the generation of the value set P(jw, Q) in the case of affine plants. The reader 
familiar with Nyquist-type analysis and design is aware of the fact that a certain range of frequencies, 
generally close to the crossover frequencies, can be specified a priori. That is, a range 2 = [wm , o*] may 
be imposed by design specifications or estimated by performing a frequency analysis of the nominal 
system under consideration. In this section, we assume that Dp(jw, q) # 0 for all q € Q and w € 2. We 
remark that if the frequency w = 0 lies in the interval (2, this assumption is not satisfied for type 1 or 
2 systems—however, these systems can be easily handled with contour indentation techniques as in the 
classical Nyquist analysis. We also observe that the assumption that P(s,q) does not have poles in the 
interval [w7, wo] simply implies that P(jw, Q) is bounded. 

To proceed with the value set construction, we first need a preliminary definition. The one-dimensional 
exposed edge e'* is a convex combination of the adjacent vertices* q' and q* of Q 


e* = agit (1—nq* 


for x € [0, 1]. Denote by E the set of all q € e* for some i, k, and 2 € [0, 1]. This set is the collection of all 
one-dimensional exposed edges of Q. 

Under our assumption OP(jw, Q) # 0, for q € Q and all w, the set P(jw, Q) is compact. Then, for fixed 
w € Q, it can be shown (see Fu, 1990) that 


OP(jw, Q) © P(jw, E) = {P(jw, q) :q € E} 


where OP(jw, Q) denotes the boundary of the value set and P(jw, E) is the image in the complex plane 
of the exposed edges (remember that we assumed OP(jw, Q) 4 0, for q € Q and all w). This says that the 
construction of the value set only requires computations involving the one-dimensional exposed edges. 
The second important fact observed in Fu (1990) is that the image of the edge e’ in the complex plane is 
an arc of a circle or a line segment. To explain this claim, in view of the affine dependence of both N(s, q) 
and D(s, q) versus q, we write the uncertain plant corresponding to the edge e“* in the form 


Np(s, qi +(1—)q*) — Np(s,q*) + XNp(s, (q! — g*)) 


ik 
ENOTES Dp(s, xqi +(1— qk) — Dp(s, q*) + XNp(s, (q’ — g*)) 


for i € [0,1]. For fixed s = jw, it follows that the mapping from the edge e“* to the complex plane is 
bilinear. Then, it is immediate to conclude that the image of each edge is an arc of a circle or a line 
segment; the center and the radius of the circle and the extreme points of the segment can be also 
computed. Even though the number of one-dimensional exposed edges of the set Q is (2°~1, the set E is 
one dimensional. Therefore, a fast computation of P(jw, E) can be easily performed and the boundary of 
the value set P(jw, Q) can be efficiently generated. 

Finally, an important extension of this approach is robustness analysis of systems with time delay. That 
is, instead of P(s, q), we consider 


_ N(s,q) 
D(s, q) 


eS 


P.(s,q) 


where t > 0 is a delay. It is immediate to see that the value set of P,(s, q) at frequency s = jw is given 
by the value set of the plant N(s, q)/D(s, q) rotated with respect to the origin of the complex plane of 
an angle tw in clockwise direction. Therefore, Criterion 7.1 still applies; see Barmish and Shi (1990) for 
further details. 


* Two vertices are adjacent if they differ for only one component. For example, in Figure 7.3 the vertices q! = (1.2, 2.2, 2.04) 


and qv = (1.2, 2.2, 3.74) are adjacent. 
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7.5.2 The DC-Electric Motor Example Revisited 


To illustrate the concepts discussed in this section, we revisit the DC-electric motor example. We take 
two uncertain parameters 

n=J; q=B, 
where q1 €[0.03,0.15] and q2 €[0.001,0.03] with nominal values J = 0.042 kg m? and B= 
0.01625 N m/rps. The remaining parameters take values K = 0.9 V/rps, L = 0.025 H, and R= 5Q. The 
voltage to angle uncertain plant is 


0.9 
0.025q1s? + (5q) + 0.025q2)s? + (0.81 + 5q2)s" 


P(s,q) = 


To proceed with robustness analysis, we first estimate the critical range of frequencies obtaining Q = 
[10, 100]. We note that the denominator of P(s, q) is nonvanishing for all q € Q in this range. Then, we 
study robust stability of the plant connected in a feedback loop with a PID compensator 


K, 
C= = Rae 


For closed-loop stability, we recall that the Nyquist criterion requires that the Nyquist plot of the 
open-loop system does not go through the point —1+ j0 and that it does encircle this point (in counter- 
clockwise direction) a number of times equal to the number of unstable poles; for example, see Horowitz 
(1963). In this specific case, setting Kp = 200, Ky = 5120, and Kp = 20, we see that the closed-loop nomi- 
nal system is stable with a phase margin } © 63.7° anda crossover frequency w, © 78.8 rad/s. Asa result of 
the analysis carried out by sweeping the frequency, it turns out that the closed-loop system is not robustly 
stable, since at the frequency w ~ 16 rad/s the value set includes the point — 1+ j0. Figure 7.4 shows the 
Nyquist plot of the nominal plant and the value set for 12 equispaced frequencies in the range (12,34). 


4 


Im 0 


FIGURE 7.4 Nyquist plot and value set. 
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Im 


FIGURE 7.5 Nyquist plot and value set. 


To robustly stabilize P(s,q), we take Ky = 2000 and the same values of Kp and Kp as before. The 
reasons for choosing this value of K7 can be explained as follows: The compensator transfer function has 
phase zero at the frequency ® = ,/K7/Kp. Thus, reducing K; from 5120 to 2000 causes a drop of @ from 
16 to 10 rad/s. This implies that the phase lead effect begins at lower frequencies “pushing” the value set 
out of the critical point — 1+ j0. Since the nominal system is stable with a phase margin $ © 63.7° and a 
crossover frequency w, * 80.8 rad/s, this new control system has nominal performance very close to the 
previous one. However, with this new PID compensator, the system becomes robustly stable. To see this, 
we generated the value sets for the same frequencies as before; see Figure 7.5. 

From this figure, we observe that P(jw, q) does not include the point —1+ j0. We conclude that the 
closed-loop system is now robustly stable; the worst-case phase margin is ) © 57.9°. 


7.6 Robustness Analysis for Affine Polynomials 


In this section, we study robustness analysis of the closed-loop polynomial 
P(s, q) = Np(s, q)Nc(s) + Dp(s, q)Dc(s) 


when the coefficients of Np(s, q) and Dp(s, q) are affine functions of q. The main goal is to provide an 
alternative criterion for polynomials instead of plants. With this approach, we do not need the nonva- 
nishing condition about Dp(s, q); furthermore, unstable pole-zero cancellations are not an issue. In this 
case, however, we lose the crucial insight given by the Nyquist plot. For fixed frequency s = jw, we define 
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the value set p(jw, Q) C Cas 
PG, Q) = {p(jo, q) : 4 € Qh. 


As in the plant case, p(jw, Q) is a set in the complex plane which moves with frequency and which 
graphically represents the uncertain polynomial. 


Criterion 7.2: Zero-Exclusion Condition for Uncertain Polynomials 


The polynomial p(s, q) is robustly stable for all q € Q if and only if p(s, q) is stable for some q € Q and 
0 Z p(jw, Q) for allw ER. 


The proof of this criterion requires elementary facts and, in particular, continuity of the roots of p(s, q) 
versus its coefficients; see Frazer and Duncan (1929). Similar to the discussion in Section 7.5, we note that 
Criterion 7.2 is easily implementable—at least whenever the value set can be efficiently generated. That is, 
given an affine polynomial family, we take any element in this family and we check its stability. This step 
is straightforward using the Routh table or any root finding routine. Then, we sweep the frequency w over 
a selected range of critical frequencies 2 = [w~, w*]. This interval can be estimated, for example, using 
some a priori information on the specific problem or by means of one of the bounds given in Marden 
(1966). If there is no intersection of p(jw, Q) with the origin of the complex plane for all w € &, then 
p(s, q) is robustly stable. 


Remark 7.1 


A very similar zero exclusion condition can be stated for more general regions D than the open left 
half-plane. Meaningful examples of D regions are the open unit disk, a shifted left half-plane and a 
damping cone”. In this case, instead of sweeping the imaginary axis, we need to discretize the boundary 
of D. 


7.6.1 Value Set Construction for Affine Polynomials 


In this section, we discuss the generation of the value set p(jw, Q). Whenever the polynomial coefficients 
are affine functions of the uncertain parameters, the value set can be easily constructed. To this end, 
two key facts are very useful. First, for fixed frequency, we note that p(jw, Q) is a two-dimensional 
convex polygon. Second, letting q', q,...,q/ denote the vertices of Q as in Section 7.4, we note that the 
vertices of the value set are a subset of the complex numbers p(j«, q!), p(jw, q”),..-» pia, q"). These two 
observations follow from the fact that, for fixed frequency, real and imaginary parts of p(s, q) are both 
affine functions of g. Then, the value set is a two-dimensional affine mapping of the set Q and its vertices 
are generated by vertices of Q. Thus, for fixed , it follows that 


p(jw, Q) = conv {p(ja, q'), p(jo, 7), ..-,p(jo, q’)} 


where conv denotes the convex hull’. The conclusion is then immediate: For fixed frequency, one can 
generate the points p(jw, q'), p(j@, q),-.-,p(jw,q") in the complex plane. The value set can be con- 
structed by taking the convex hull of these points—this can be readily done with 2D graphics. From the 
computational point of view, we observe that the number of edges of the polygon is at most 2@ at each 
frequency. This follows from the observations that any edge of the value set is the image of an exposed 
edge of Q. In addition, parallel edges of Q are mapped into parallel edges in the complex plane and the 
edges of Q have only @ distinct directions. These facts can be used to efficiently compute p(jw, Q). We 
now provide an example which illustrates the value set generation. 


* A damping cone is a subset of the complex plane defined as {s : Re(s) < —al|Im(s)|} for a > 0. 


The convex hull conv S of a set S is the smallest convex set containing S. 
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7.6.2 Example of Value Set Generation 


Using the same data as in the example of Section 7.5.2 and a PID controller with gains Kp = 200, 
K; = 5120, and Kp = 20, we study the closed-loop polynomial 


p(s, q) = 0.025q1s* + 8°(5q) + 0.025q2) + s?(5q2 + 18.81) + 180s + 4608. 


Robustness analysis is performed for 29 equispaced frequencies in the range (2,30). Figure 7.6 shows the 
polygonality of the value set and zero inclusion for w ~ 16 rad/s which demonstrates instability. This 
conclusion is in agreement with that previously obtained in Section 7.5.2. 


7.6.3 Interval Polynomials: Kharitonov’s Theorem and Value Set Geometry 


In the special case of interval polynomials, robustness analysis can be greatly facilitated via Kharitonov’s 
Theorem (Kharitonov, 1978). We now recall this result. Given an interval polynomial 


p(s, q) = qo + 41s + qos” + +++ +4ns” 


of order n (i.e., dn 4 0) and bounds [q; , qi | for each coefficient q;, define the following four polynomials: 


Pils) = qo tap Sts taqys tags +qss tggst+aq7s’ tes 


jg tarstays tags tags +aq5se tags tays’ ++ °3 


p(s) = qq +q stags +43 +q; st +45 Ce ee eee 


=qo taqfstaye tase +a tags tags’ +aq7s’+--- 
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FIGURE 7.6 Value set plot. 
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Then, p(s, q) is stable for all q € Q if and only if the four Kharitonov polynomials p;(s), p2(s), p3(s) and 
pa(s) are stable. To provide a geometrical interpretation of this result, we observe that the value set for 
fixed frequency s = jw is a rectangle with level edges parallel to real and imaginary axis. The four vertices 
of this set are the complex numbers p1 (jw), p2(jw), p3(jw), and p4(jw); see Dasgupta (1988). If the four 
Kharitonov polynomials are stable, due to the classical Mikhailov’s criterion (Mikhailov, 1938), their 
phase is strictly increasing for « increasing. In turn, this implies that the level rectangular value set moves 
in a counterclockwise direction around the origin of the complex plane. Next, we argue that the strictly 
increasing phase of the vertices and the parallelism of the four edges of the value set with real or imaginary 
axis guarantee that the origin does not lie on the boundary of the value set. By continuity, we conclude that 
the origin is outside the value set and the zero exclusion condition is satisfied; see Minnichelli et al. (1989). 


7.6.4 From Robust Stability to Robust Performance 


In this section, we point out the important fact that the polynomial approach discussed in this chapter can 
be also used for robust performance. To explain, we take an uncertain plant with affine uncertainty and 
we show how to compute the largest peak of the Bode plot magnitude for all q € Q—that is, the worst-case 
Hoo norm. Formally, for a stable strictly proper plant P(s, q), we define 


max ||P(s, q)|loo = max sup |P(jo, q)|. 
qEQ qEQ w 
Given a performance level y > 0, then 
max ||P(s, q)|loo < Y 
9EQ 
if and only if 


Np(jo, q) 
Dp(ja, q) 


for all mw > 0 and q € Q. Since P(jw, q) > 0 for w — on, this is equivalent to check if the zero-exclusion 
condition 


Np(jw, q) — yDe(jw, ge? £0 


is satisfied for all w € R,q € Q and o € [0, 27]. In turn, this implies that the polynomial with complex 
coefficients 


po(s,q) = Np(s, q) — yDp(s, qe!” 


has roots in the open left half-plane for all q € Qand  € [0, 21]. Clearly, for fixed  € [0, 21], Criterion 7.2 
can be readily used; however, since p(s, q) has complex coefficients, it should be necessarily checked for 
all w € R, including negative frequencies. 


7.6.5 Algebraic Criteria for Robust Stability 


If the uncertain polynomial under consideration is affine, the well-known Edge Theorem applies (Bartlett 
et al., 1988). This algebraic criterion is alternative to the frequency domain approach studied in this 
section. Roughly speaking, this result says that an affine polynomial family is robustly stable if and only 
if all the polynomials associated with the one-dimensional exposed edges of the set Q are stable. Even 
though this result is of algebraic nature, it can be explained by means of value set arguments. For affine 
polynomials and for fixed w, we have already observed in Section 7.6.3 that the boundary of the value set 
p(jo, Q) is the image of the one-dimensional exposed edges of Q. Thus, to guarantee the zero-exclusion 
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condition, we need to guarantee that all edge polynomials are nonvanishing for all w € R—otherwise an 
instability occurs. We conclude that stability detection for affine polynomials requires the solution of a 
number of one-dimensional stability problems. Each of these problems can be stated as follows: Given 
polynomials po(s) and p;(s) of order n and m < n, respectively, we need to study the stability of 


p(s, ) = po(s) + rpi(s) 


for all } € [0,1]. A problem of great interest is to ascertain when the robust stability of p(s, ) can be 
deduced from the stability of the extreme polynomials p(s, 0) and p(s, 1). This problem can be formulated 
in more general terms: To construct classes of uncertain polynomials for which the stability of the vertex 
polynomials (or a subset of them) implies stability of the family. Clearly, the edges associated with an 
interval polynomial is one such class. Another important example is given by the edges of the closed-loop 
polynomial of a control system consisting of a first-order compensator and an interval plant; see Barmish 
et al. (1992). Finally, see Rantzer (1992) for generalizations and for the concept of convex directions 
polynomials. 


7.6.6 Further Extensions: The Spectral Set 


In some cases, it is of interest to generate the entire root location of a family of polynomials. This leads 
to the concept of spectral set; see Barmish and Tempo (1991). Given a polynomial p(s, q), we define the 
spectral set as 

o ={s €C: p(s,q) =0 for some q€ Q}. 


The construction of this set is quite easy for affine polynomials. Basically, the key idea can be explained 
as follows: For fixed s € C, checking if s isa member of the spectral set can be accomplished by means of the 
zero-exclusion condition; see also Remark 7.6.2. Next, it is easy to compute a bounded root confinement 
region 6 > o; for example, see Marden (1966). Then, the construction of the spectral set o amounts to a 
two-dimensional gridding of 6 and, for each grid point, checking zero exclusion. 

The spectral set concept can be further extended to control systems consisting of a plant P(s, q) with a 
feedback gain Kp which needs tuning. In this case, we deal with the so-called robust root locus (Barmish 
and Tempo, 1990)—that is, the generation of all the roots of the closed-loop polynomial when Kp ranges 
in a given interval. To illustrate, we consider the same data as in Section 7.6.3 and a feedback gain Kp, 
thus obtaining the closed loop polynomial 


p(s, q) = 0.025q1s° + (5q1 +0.025q2)s” + (0.81 + 5q2)s + 0.9Kp, 


where q; € [0.03, 0.15] and q2 € [0.001, 0.03]. In Figure 7.7, we show the spectral set for Kp = 200. Actu- 
ally, only the portion associated with the dominant roots is visible since the spectral set, obviously, includes 
a real root in the interval [—200.58, —200.12] which is out of the plot. 


7.7 Multiaffine Uncertainty Structures 


In this section, we discuss the generation of the value set for polynomials with more general uncertainty 
structures than affine. In particular, we study the case when the polynomial coefficients are multiaffine 
functions of the uncertain parameters. Besides the motivations provided in Section 7.3, we recall that 
this uncertainty structure is quite important for a number of reasons. For example, consider a linear 
state-space system of the form x(t) = A(q)x(t) where each entry of the matrix A(q) € R”™*”™ lies in a 
bounded interval, that is, it is an interval matrix. Then, the characteristic polynomial required for stability 
considerations has a multiaffine uncertainty structure. 

In the case of multiaffine uncertainty, the value set is generally not convex and its construction cannot 
be easily performed. However, we can easily generate a “tight” convex approximation of p(jw, Q)—this 
approximation being its convex hull conv p(jw, Q). More precisely, the following fact, called the Mapping 
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FIGURE 7.7 Spectral set. 


Theorem, holds: The convex hull of the value set conv p(jw, Q) is given by the convex hull of the vertex 
polynomials p(j«, q!), p(jw, q?),...,p(jw, q"). In other words, the parts of the boundary of p(j«, Q) 
which are not line segments are always contained inside this convex hull. Clearly, if conv p(jw, Q) is used 
instead of p(jw, Q) for robustness analysis through zero exclusion, only a sufficient condition is obtained. 
That is, if the origin of the complex plane lies inside the convex hull, we do not know if it is also inside 
the value set. We now formally state the Mapping Theorem; see Zadeh and Desoer (1963). 


Theorem 7.1: Mapping Theorem 
For fixed frequency w € R, 
conv p(jw, Q) = conv {p(jw, q'), p(jw, q7),---,p(jo, q“)}. 


With regard to applicability and usefulness of this result, comments very similar to those made in 
Section 7.7 about the construction of the value set for affine polynomials can be stated. Figure 7.8 
illustrates the Mapping Theorem for the polynomial 


p(s, q) =s° + (qo +4q3 + 2q142)s* + (4q293 + 419294)8 + 93 + 24193 — 91.92(g4 — 0.5) 


with four uncertain parameters q1, q2, q3, and q4 each bounded in the interval [0, 1] and frequency w = 1 
rad/s. The “true” value set shown in this figure is obtained via random generation of 10,000 samples 
uniformly distributed in the set Q. 
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FIGURE 7.8 Value set and its convex hull. 


7.8 General Uncertainty Structures and Controller Synthesis 


For general uncertainty structures there is no analytical tool that enables us to construct the value set. For 
systems with a very limited number of uncertain parameters, a brute force approach, as in the example 
of Theorem 7.1, can be taken by simply gridding the set Q with a sufficient number of points. With 
this procedure, one can easily obtain a “cloud” in the complex plane which approximates the value set. 
Obviously, this method is not practical for a large number of uncertain parameters and provides no 
guarantee that robustness is satisfied outside the grid points. 

Several attempts in the literature aimed at solving this general problem. Among the pioneering 
contributions along this line we recall the parameter space approach (Ackermann, 1980), techniques 
for the multivariable gain margin computation (de Gaston and Safonov, 1988) and (Sideris and Sanchez 
Pena, 1989) and the geometric programming approach (Vicino et al., 1990). 

As far as the robust synthesis problem is concerned, it is worth mentioning that the methods described 
in this chapter are suitable for synthesis in a trial-and-error fashion. A practical use of the techniques can 
be summarized in the following two steps (as illustrated in Section 7.5.2): 


¢ Synthesize a controller for the nominal plant with any proper technique. 
¢ Perform robustness analysis taking into account the uncertainties and go back to the previous step 
if necessary. 


In the special case of low complexity compensators, the two steps can be combined. This is the case of 
PID synthesis for which efficient robust design tools have been proposed (Ho et al., 2000). 

To facilitate robustness analysis for general parametric uncertainty structures and for nonparametric 
uncertainty, different approaches have been proposed in the literature. Among the others, we recall the 
\ approach (Zhou et al., 1996; Sanchez Pena and Sznaier, 1998) and the probabilistic methods (Tempo 
et al., 2005). These approaches can be also used for designing a controller, but 1-synthesis is based on a 
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procedure which is not guaranteed to converge to a global solution and probabilistic methods provide a 
controller which satisfies the required specification only with a given probability. However, both methods 
are useful tools for many robust control problems. 

Asa final remark, we emphasize that in this chapter we considered only uncertainties which are constant 
in time. It is known that parameter variations or, even worse, switching, may have a destabilizing effect. 
There are, indeed, examples of very simple systems that are (Hurwitz) stable for any fixed value of the 
parameters but can be destabilized by parameter variations. A class of systems which has been investigated 
is that of Linear Parameter-Varying (LPV) systems. For these systems, it has been established that the 
Lyapunov approach is crucial. For an overview of the problem of robustness of LPV systems and the link 
with switching systems, the reader is referred to Blanchini and Miani (2008). 
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8.1 Modeling MIMO Linear Time-Invariant Systems 
in Terms of Transfer Function Matrices 


Any multivariable linear time invariant (LTI) system is uniquely described by its impulse response 
matrix. The Laplace transform of this matrix function of time gives rise to the system’s transfer function 
matrix (TFM). We assume that all systems in the sequel have TFM representations; frequency response 
analysis is always applied to TFM descriptions of systems. For the case of finite dimensional LTI systems, 
when we have a state-space description of the system, closed-form evaluation of the TFM is particularly 
easy. This is discussed in Example 8.1. More generally, infinite-dimensional LTI systems also have TFM 
representations, although closed-form evaluation of the TFM of these systems is less straightforward. 
Stability (in whatever sense) is presumed; without stability we cannot talk about the steady-state response 
of a system to sinusoidal inputs. For such systems, the TFM can be measured by means of sinusoidal 
inputs. We will not discuss the use of frequency-domain techniques in robustness analysis. 


8-1 
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Example 8.1: Finite-Dimensional LTI Systems 


Suppose that we have a finite-dimensional LTI system G, with the following state equations 
x(t) = Ax(t) + Bu(t), 
y(t) = Cx(t) + Dutt). 

Taking the Laplace transform of the state equation, we see that, 
x(s) = (sl — A)” 'Bu(s) 


where we have abused notation slightly with x(s) and u(s) representing the Laplace transforms of 
x(t) and u(t), respectively. Similarly, 


y(s) = [C(s! — A)~'B + D]u(s) = G(s)u(s). 


The matrix quantity G(s) is the transfer function matrix associated with the system G. 


As a matter of convention, we will refer to the input of the system generically as u. The more specific 
notation u(t) and u(s) will be used in reference to u represented as time-domain and frequency-domain 
(Laplace-transform) signals, respectively. A similar convention applies to the output, y. We will assume 
that u is a vector with m components and that the output y is a vector with p components. This makes the 
TEM a pxm matrix. To make this explicit, 


giuls) +++ gim(s) 
Gs)=] : (8.1) 

Bpils) +++ Spm(s) 
u(s) = [y(s),+++ , Um(s)]" (8.2) 
y(s) = [n(s),+++ yp (s)I" (8.3) 

Componentwise, 
OL OV k=1,...,p. (8.4) 
j=l 


As Laplace transforms, u(s), y(s), and G(s) are generally complex-valued quantities. In more formal 
mathematical notation, u(s) € C™, y(s) € C?, and G(s) € C?*™. 


8.2 Frequency Response for MIMO Plants 


In discussing the frequency response of LTI systems, we focus our attention on systems which are strictly 
stable. This allows us to envision applying sinusoidal inputs to the system and measuring steady-state 
output signals which are appropriately scaled and phase-shifted sinusoids of the same frequency. Because 
we are dealing with MIMO systems, there are now additional factors which affect the nature of the 
frequency response, particularly the relative magnitude and phase of each of the components of the input 
vector u. These considerations will be discussed in detail below. 
Suppose that we have in mind a complex exponential input, as below, 

u(t) = inet! (8.5) 
where i = (iij,..., iim)! is a fixed (complex) vector in C”. Note that by allowing u to be complex, the 
individual components of u(t) can have different phases relative to one another. 
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Example 8.2: 


Suppose that m = 2 and that u = ((1+4+-/1), (1 —j1))', then, 


— (O4ID) jot ( V2eF/4 \ oe (V2eiotra/4) 
wo=(( ae Shee elie Vieilot—x/4) 


Thus, the two components of u(t) are phase shifted by 1/2 radians (or 90 degrees). 


Suppose that this input is applied to our stable LTI system G (of compatible dimension). We know from 
elementary linear systems theory that each component of the output of G can be expressed in terms of 
G’s frequency response G( jw). (We obtain the frequency response from G(s), literally, by setting s = jw.) 
Thus, at steady state, 


m 


yelt) = Yo gg(jo)ijel™; k=1,...,p. (8.6) 
j=l 


We may now express the vector output y(f) at steady state as follows: 


y(t) = jel", P=(n,....Fp)” € CP, (8.7) 
where 
Vk = Gor ea tener |» (8.8) 
j=l 
Putting all of this together, 
y = G(jo)u. (8.9) 


Just as in the SISO case, the MIMO frequency response G(jw) provides a convenient means of 
computing the output of an LTI system driven by a complex exponential input. Analysis of the fre- 
quency response, however, is now complicated by the fact that G( jw) is a matrix quantity. A simple way 
is needed to characterize the “size” of the frequency response as a function of w. The effect of G( jw) on 
complex exponential input signals depends on the direction of u € C™, including the relative phase of the 
components. In fact, a whole range of “sizes” of G( jw) exists, depending on the directional nature of wu. 
Our characterization of size should thus provide both upper and lower bounds on the magnitude gain of 
the frequency response matrix. The mathematical tool we need here is the Singular Value Decomposition 
(SVD) discussed briefly in the following section. 


8.3 Mathematical Detour 


We present here some mathematical definitions and basic results from linear algebra germane to our 
discussion of MIMO frequency response analysis. An excellent reference for this material can be found 
in [5]. 
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8.3.1 Introduction to Complex Vectors and Complex Matrices 


Given a complex-valued column vector x € C”, we may express x in terms of its real and imaginary 
components, 
x] ay + jb 
x=]: ]= : =a-+jb 
Xn An + jOn 


where a and b are both purely real-valued vectors in C”. We define the row vector x"! as the complex- 


conjugate transpose of x, i.e., 
H * 


xt = (x*,...,x*) = al —jo". 
The superscript asterix above denotes the complex-conjugate operation. 

If we have two vectors x and y, both elements of C”, then the inner product of x and y is given by the 
(complex) scalar x" y. Two vectors x and y in C", which satisfy xy = 0, are said to be orthogonal. 

The Euclidean norm of x, denoted by ||x||2, is given by the square root of the inner product of x with 
itself, 


IIxlla=Vx!x = VaTatblb= |S (a? +b/). 


i=1 


It is important to note that || - ||z is a real scalar function chosen arbitrarily to reflect the “size” of vectors in 
C”. (Itis true that, as anorm, || - ||2 has to satisfy certain mathematical requirements, particularly positivity, 
scaling, and the triangle inequality. Aside from this, our definition of || - ||2 is arbitrary.) Because all of 
the components of x are taken into account simultaneously, the value (and interpretation) of ||x||2 will 
depend on the units in which the components of x are expressed. 


Example 8.3: 


Suppose that we are dealing with a high-power (AC) electronic device and that the state of the device 
is determined by a vector x € C2 made up of phased voltages at two distinct points in the circuitry. 
Suppose first that both quantities are expressed in terms of kilovolts (kV). For example, 


x =(1+j2,2—j3)' kv 
then, 


IIx]? = [0 —j2), (2 + J3)10 + j2), (2 —f3) 7 
=(1+4)+(44+9) 
=18 


If, however, the first component is expressed in terms of Volts (V), then 


IIx\I2? = [1000 — j2000), (2 + j3)][(1000 + j2000), (2 — j3)]", 


= (10° +4 x 10°) + (4 +9) 
= 5000013, 


which is a far cry from what we had before! Note that this is not an entirely unreasonable example. 
In general, the components of x can consist of entirely different types of physical quantities, such 
as voltage, current, pressure, concentration, etc. The choice of units is arbitrary and will have an 
important impact on the “size” of x when measured in terms of the Euclidean norm. 
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A complex-valued matrix M € C?*™ is a matrix whose individual components are complex valued. 
Since M is complex valued, we can express it as the sum of its real and imaginary matrix parts, i-e., 
M = A-+43B, where A and B are both purely real-valued matrices in C?*™. 

We define the Hermitian of a complex matrix M as the complex-conjugate transpose of M, that is, MY 
is computed by taking the complex conjugate of the transpose of M. Mathematically, 


M# — At — jpt (8.10) 
The following will play an important role in the next subsection. 


8.3.1.1 Important Fact 


Both MM and MM" have eigenvalues that are purely real valued and nonnegative. Moreover, their 
nonzero eigenvalues are identical even though M?M 4 MM#. 


Example 8.4: 
Let 
m=[! oa 
-j 2+) 
Then, 
H_| 1 J 
. =|; =] 2 
Hy —| 2 J 
wwe[ 2, 
and 


y_[ 3 34/2 
oe =[32 0 iad 


Although MM and MMH are clearly not equal, a simple calculation easily reveals that both products 
have the same characteristic polynomial, 
det(xJ — M4 mM) = det(.l — MM") = 2 — 92 +5 


This implies that MM and MM" share the same eigenvalues. 


A complex-valued matrix M is called Hermitian if M = M". A nonsingular, complex-valued matrix 
is called unitary if M~! = M™. Stated another way, a complex-valued matrix M is unitary if its column- 
vectors are mutually orthonormal. 

The spectral norm of a matrix M € C?*™, denoted ||M|l2, is defined by 


|M|l2= max ||Mx|l2 (8.11) 

xll2==1 
The best way to interpret this definition is to imagine the vector x rotating on the surface of the unit 
hypersphere in C”, generating the vector Mx in C?. The size of Mx, i.e., its Euclidean norm, will depend 


on the direction of x. For some direction of x, the vector Mx will attain its maximum value. This value is 
equal to the spectral norm of M. 


8.3.2 The Singular Value Decomposition 


In this subsection we give a quick introduction to the singular value decomposition. This will be an 
essential tool in analyzing MIMO frequency response. For more details, the reader is referred to [5]. 
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8.3.2.1 The Singular Values of a Matrix 


Suppose that M is a p x m matrix, real or complex. Assume that the rank of M is k. We associate with M 
a total of k positive constants, denoted o;(M), or simply 0;,i=1,...,k. These are the singular values of 
M, computed as the positive square roots of the nonzero eigenvalues of either M4 M or MM", that is, 


o;(M) = /;(MEM) = /i(MM#) >0, i=1,...,k (8.12) 


where },(-) is a shorthand notation for “the ith nonzero eigenvalue of”. Note that the matrices MEM 
and MM" may have one or more zero valued eigenvalues in addition to the ones used to compute the 
singular values of M. It is common to index and rank the singular values as follows: 


01(M) > 02(M) => --- > 0, (M) > 0 (8.13) 


The largest singular value of M, denoted oy4x(M), is thus equal to 0j(M). Similarly, omin(M) = 0;(M). 

While it is tricky, in general, to compute the eigenvalues of a matrix numerically, reliable and effi- 
cient techniques for computing singular values are available in commercial software packages, such as 
MATLAB®, 


8.3.2.2 The Singular Value Decomposition 


The SVD is analogous to matrix diagonalization. It allows one to write the matrix M in terms of its singular 
values and involves the definition of special directions in both the range and domain spaces of M. 

To begin the definition of the SVD, we use the k nonzero singular values 0; of M, computed above. First 
form a square matrix with the k singular values along the main diagonal. Next, add rows and columns of 
zeros until the resulting matrix X is p x m. Thus, 


O1 0 -.. 0 
0 02 0 
SS ses Sas Gae- as kx(m—k) (8.14) 
0 O +++ OK 
O(p—k) xk O(p—k)x(m—k) 


By convention, assume that 


Theorem 8.1: Singular Value Decomposition 


A p x p unitary matrix U (with U = U™) and an m x m unitary matrix V (with V = V™) exist so that 
M=UnV" y=UMMV (8.15) 
The p-dimensional column vectors u;,i=1,...,p, of the unitary matrix U are called the left singular 


vectors of M. The m-dimensional column vectors v;,i = 1,...,m of V are called the right singular vectors 
of M. Thus, we can visualize, 


U=[u uz ... up), V=[v v2... Vm] 
Since U and V are each unitary, 
ul up = Six, i,k=1,...,p 
vi vy = Bigs ijik=1,...,m 


where 8;, is the Kronecker delta. Because the left and right singular vectors are linearly independent, 
they can serve as basis vectors for C? and C”, respectively. Moreover, the left and right singular vectors 
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associated with the (nonzero) singular values of M span the range and left-null spaces of the matrix M, 
respectively. Finally, a simple calculation shows that the left singular vectors of M are the normalized 
right eigenvectors of the p x p matrix MM". Similarly, the right singular vectors of M are the normalized 
left eigenvectors of the p x p matrix M"M. 


8.3.2.3 Some Properties of Singular Values 


We list here some important properties of singular values. We leave the proofs to the reader. Some of the 
properties require that the matrix be square and nonsingular. 


1. Omax(M) = max|x\.=1 || Mx|l2 = ||Mll2 = 


Omin(M) = minyx|=1 || Mx||2 = Test = Stu: 
oj(M) —1<o0;)7+M) <o)(M)+1, i=1,...,k. 
0j(aM) = |alo;(M) for alla € C,i=1,...,k. 
Omax(My + M2) < Omax(M1) + Omax (M2). 
Omax(M1M2) < Omax(M1) - Omax(M2). 


1 
omin(M~") : 


AM Pw Ny 


Property 1 indicates that maximum singular value 0;4,(M) is identical to the spectral norm of M. Thus, 
Properties 5 and 6 are restatements of the triangle inequality and submultiplicative property, respectively. 


8.3.2.4 The SVD and Finite Dimensional Linear Transformations 


We shall now present some geometric interpretations of the SVD result. Consider the linear 
transformation 
y=Mu, ueC™, yeC?. (8.16) 


Let M have the singular value decomposition discussed above, that is, M = ULV". It may help the reader 
to think of u as the input to a static system M with output y. From the SVD of M, 


y =Mu=UEV"u. 


Suppose we choose u to be one of the right singular vectors, say vj, of M. Let y; denote the resulting 
“output” vector. Then, 
yj = My, = ULV". 


Because the right singular vectors of M are orthonormal, 
H T 
VOM = (058590510) 2.50). 5 


where the ith component only takes on the value of 1. In view of the special structure of the matrix of 
singular values ©, 


EV4y; = (0,...,0,0;,0,...0)' 


where, again, only the ith component is potentially nonzero. Thus, finally, 
Nix UxV" y; = Ojlj. (8.17) 


Equation 8.17 interprets the unique relationship between singular values and singular vectors. In the 
context of M as a “static” system, when the input u is equal to a right singular vector v;, the output 
direction is fixed by the corresponding left singular vector u;. Keeping in mind that both u; and v; have 
unit magnitudes (in the Euclidean sense), the amplification (or attenuation) of the input is measured by 
the associated singular value oj. If we choose u = vj, where i > k, then the corresponding output vector is 
zero because the matrix is not full rank and there are no more (nonzero) singular values. 

Because Equation 8.17 holds for i=1,...,k, it is true in particular for the maximum and minimum 
singular values and associated singular vectors. By abuse of notation, we shall refer to these left and right 
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A Uy AY, 


FIGURE 8.1 Visualization of SVD quantities. 


singular vectors as maximum and minimum singular vectors, respectively, and use the subscripts “max” 
and “min” to distinguish them. Within the context of “static” systems, inputs along the maximum right 
singular vector generate the /argest output along the direction of the maximum left singular vector. Similar 
comments apply to the case where inputs are in the direction of the minimum left singular vector. 


Example 8.5: The Case Where M Is Real and 2 x 2 


Let us suppose that M is a real-valued, nonsingular matrix mapping u € R2 to y = Mu € R2. Let us 

suppose further that u rotates on the circumference of the unit circle. The image of u under the 

transformation of M will then trace an ellipse in the (output) plane, as illustrated in Figure 8.1. 
Because M is a real, nonsingular, 2 x 2 matrix, the SVD analysis will give 


Oo 0 
r= | U = [Umax Umin] VY = [Vmax Vmin] 
Omin 


where &, U, and V are all real valued. 


Suppose that when u equals the vector OA, the output y is the vector OA’. Suppose further that, 
Y =Ymax = OmaxUmax- Thus, the maximum right singular vector Vax equals the (unit length) vector OA, 
and the maximum left singular vector uma, equals the (unit length) vector OA”. Moreover, the maximum 
singular value, Oma, equals the length of the vector OA’. 

Similarly, suppose that when u equals the vector OB, the output y is the vector OB’. Suppose further that, 
¥ =Ymin = OminUmin. Thus, the minimum right singular vector vi, equals the (unit length) vector OB, 
and the minimum left singular vector umin equals the (unit length) vector OB”. Moreover, the minimum 
singular value, Oj, equals the length of the vector OB’. 

Notice in Figure 8.1 that the left singular vectors are normal to each other, as are the right singular 
vectors. 

As the minimum singular value decreases, so does the semiminor axis of the ellipse. As this happens, 
the ellipse becomes more and more elongated. In the limit, as oi, — 0, the ellipse degenerates into a 
straight line segment, and the matrix M becomes singular. In this limiting case, there are directions in the 
output space that we cannot achieve. 

If the matrix M were a 3 x 3 real nonsingular matrix, then we could draw a similar diagram, illus- 
trating the unit sphere mapping into a three-dimensional ellipsoid. Unfortunately, diagrams for higher 
dimensional matrices are impossible. Similarly, diagrams for complex matrices (even 2 x 2 matrices) are 
impossible, because we need a plane to represent each complex number. 
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Using these geometric interpretations of SVD quantities, it is possible to be precise about the meaning 
of the “size” of a real or complex matrix. From an intuitive point of view, if we consider the “system” 
y = Mu, and if we restrict the input vector u to have unit length, then 


1. The matrix M is “large” if |yll2 >> 1, independent of the direction of the unit input vector u. 
2. The matrix M is “small” if ||y||z2 « 1, independent of the direction of the unit input vector u. 


If we accept these definitions, then we can quantify size as follows: 


1. The matrix M is “large” if its minimum singular value is large, ie., Omin(M) > 1. 
2. The matrix M is “small” if its maximum singular value is small, i.e., Omax(M) < 1. 


8.3.2.5 More Analytical Insights 


Once we have computed an SVD for a matrix M, in y = Mu, then we can compute many other important 
quantities. In particular, suppose that M is m x m and nonsingular. It follows that M has m nonzero 
singular values. We saw earlier (in Equation 8.17) that 


Vi = OjUi 


when u = v;,i=1,...,m. Because the left singular vectors are orthonormal, they form a basis for the 
m-dimensional input space, so that we can write any (input) vector in C” as a linear combination of the 
vis. For example, let u be given as follows: 


Uu=YViVi + Y2V2 +++ + VmVm 
where the y; are real or complex scalars. From the linearity of the transformation M, 
y = Mu = y101 U1 + 20242 + ++ > + ¥mOmVm- 


Using the SVD, we can also gain insight on the inverse transformation u = M~'y. From the SVD 
theorem, we know that M = UXV". Using the fact that U and V are unitary, M~! = VD~!U". Notice 


that 
1 1 1 
Bt =diag | =) 2 ral 
0] 02 Om 
Thus, if 
y = 8u + 82u2 + +++ + 8mm 
then ‘ i ‘ 
u= My =8)—vy +8.—v2 ++ ++ +8 —Vin 
O1 02 Om 


This implies that the information in the SVD of M can be used to solve systems of linear equations without 
computing the inverse of M. 


8.4 The SVD and MIMO Frequency Response Analysis 


We now return to our discussion of MIMO frequency response with the full power of the SVD at our 
disposal. Once again, we shall focus our attention on the transfer function matrix (TFM) description of 
the strictly stable LTI system G(s). As before, we will assume that G has m inputs and p outputs, making 
G(s) a p x m matrix. In general we shall assume that p > m, so that, unless the rank k of G(s) becomes 
less than m, the response of the system to a non-zero input is always non-zero. 

Recall that if the input vector signal u(t) is a complex exponential of the form u(t) = ie!”!, with i 
fixed in C”, then at steady state, the output vector y(t) will also be a complex exponential function, 
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y(t) = yes", for some y € CP. Recall, also, that the complex vectors i and j are related by G(s) evaluated 
at s = ja, that is, 
y = G(jow)u. 


It is important to note that G(jw) is a complex matrix that changes with frequency w. For any given 
fixed frequency, we can calculate the SVD of G(jw): 


G(jo) = U(jw)X(o)V" (jo) 


Note that, in general, all of the factors in the SVD of G(jw) are explicitly dependent on omega: 


1. The matrix X(w) is a p x m matrix whose main diagonal is composed of the singular values of 
G(jo), 


Omax(@) = 01(@), 62(w),...; OK, (w) = Omin(@) 


where k,, is the rank of G( jo). 

2. The matrix U(jw) is an m x m complex-valued matrix whose column vectors {u;(j@)} are the left 
singular vectors of G( jo). 

3. The matrix V(jw) is a p x p complex-valued matrix whose column vectors {v;(j)} are the right 
singular vectors of G( jo). 


8.4.1 Singular Value Plots (SV Plots) 


Once we calculate the maximum and minimum singular values of G(jw) for a range of frequencies w, 
we can plot them together on a Bode plot (decibels versus rad/sec in log-log scale). Figure 8.2 shows a 
hypothetical SV plot. 

With the proper interpretation, the SV plot can provide valuable information about the properties 
of the MIMO system G. In particular, it quantifies the “gain-band” of the plant at each frequency, and 
shows how this changes with frequency. It is a natural generalization of the information contained in the 
classical Bode magnitude plot for SISO plants. One main difference here is that, in the multivariable case, 
this “gain-band” is described by two curves, not one. 

It is crucial to interpret the information contained in the SV plot correctly. At each frequency w we 
assume that the input is a unit complex exponential, u(t) = ie/*. Then, assuming that we have reached 
steady state, we know that the output is also a complex exponential with the same frequency, y(t) = ye", 


w(rad/s) 
> 


0 dB 
OninGU w) 


FIGURE 8.2 A hypothetical SV plot. 
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where y = G(jw)u. The magnitude ||||2 of the output complex exponential thus depends on the direction 
of the input as well as on the frequency w. Now, by looking at an SV plot, we can say that, at a given 
frequency: 


1. The largest output size is ||¥||2,max = OmaxG(jo), for ||u||2 = 1. 
2. The smallest output size is ||V||2,min = OminG(jo), for ||ul|2 = 1. 


This allows us to discuss qualitatively the size of the plant gain as a function of frequency: 


1. The plant has large gain at w if OminG( jo) > 1. 
2. The plant has small gain at w if OmaxG(jo) « 1. 


8.4.2 Computing Directional Information 


In addition to computing system gain as a function of frequency, we can also use the SVD to compute 
“directional information” about the system. In particular, we can compute the direction of maximum 
and minimum amplification of the unit, real-valued sinusoidal input. In the following, we present a step- 
by-step methodology for maximum amplification direction analysis. Minimum amplification direction 
analysis is completely analogous, and will therefore not be presented explicitly. 


8.4.2.1 Maximum Amplification Direction Analysis 


1. Select a specific frequency w. 

2. Compute the SVD of G(jm), ie, find U(w),U(jw), and V(jw) such that G(jw)= 
U( jo)d(w)V# (jw) where U and V are unitary and » is the matrix of singular values. 

3. In particular, find the maximum singular value oy, (@) of G( ja). 

4. Find the maximum right singular vector vg,(@). This is the first column of the matrix V(ja) 
found in the SVD. Note that Vjnq.(@) is a complex vector with m elements. Write the elements of 
Vmax(@) in polar form, ie., 


[Vmax(@)]i = lajle/¥, i=1,2,...,m. 


Notice that a; and wy; are really functions of w; we suppress this frequency dependence for clarity. 
5. Find the maximum left singular vector ujqx(@). This is the first column of the matrix U( ja) found 
in the SVD. Note that umax(w) is a complex vector with p elements. Write the elements of Umax (@) 
in polar form, i.e., 
[Umax (@)]i = [biles*, i=1,2,...,p. 


Notice that b; and ; are functions of w; we suppress this frequency dependence for clarity. 

6. We are now in a position to construct the real sinusoidal input signals that correspond to the 
direction of maximum amplification and to predict the output sinusoids that are expected at 
steady state. The input vector u(t) is defined componentwise by 


uj(t) = |aj|sin(@t+ ij), i=1,2,...,m 


where the parameters a; and yj; are those determined above. Note that the amplitude and phase 
of each component sinusoid is distinct. We can utilize the implications of the SVD to predict the 
steady-state output sinusoids as 


Vi(t) = Omax(@)|bj| sin(wt+;), i=1,2,...,p. 


Notice that all parameters needed to specify the output sinusoids are already available from 
the SVD. 
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When we talk about the “directions of maximum amplification,” we mean input sinusoids of the form 
described above with very precise magnitude and phase relations to one another. The resulting output 
sinusoids also have very precise magnitude and phase relations, all as given in the SVD of G( jw). Once 
again, a completely analogous approach can be taken to compute the minimum amplification direction 
associated with G( jw). 

It is important to remember that the columns of U(jw) and V(jw) are orthonormal. This means we 
can express any sinusoidal input vector as a linear combination of the right singular vectors of G( jw) at a 
particular value of w. The corresponding output sinusoidal vector will be a linear combination of the left 
singular vectors, after being scaled by the appropriate singular values. 

Finally, because we measure system “size” in terms of the ratio of output Euclidean norm to input 
Euclidean norm, the “size” of the system is heavily dependent on the units of the input and output 
variables. 


8.5 Frequency Response Analysis of MIMO Feedback Systems 


In this section, we look at frequency domain-analysis for various control system configurations. We will 
pay particular attention to the classical unity-feedback configuration, where the variables to be controlled 
are used as feedback. Next we will look at a broader class of control system configurations relevant for 
some of the more modern controller design methodologies such as Hoo and 1; synthesis, as well as in 
robustness analysis and synthesis. MIMO frequency-domain analysis as discussed above will be pivotal 
throughout. 


8.5.1 Classical Unity-Feedback Systems 


Consider the unity-feedback system in the block diagram of Figure 8.3. Recall that the Joop transfer 
function matrix is defined as 
T(s) = G(s)K(s) 


The sensitivity S(s) and complementary-sensitivity C(s) transfer function matrices are, respectively, 
defined as 


S(s) =U+T(s)"! 
C(s) = [1+ T(s)]-'G(s)K(s) = S(s)T(s) 


With these definitions, 
e(s) = S(s)[r(s) — d(s)] + C(s)n(s) (8.18) 


The objective in control system design is to keep the error signal e “small”. This means the transfer 
from the various disturbances to e must be small. Because it is always true that S(s) + C(s) =I, there is a 


n(s) d(s) 


(3) el) il hee ae 6) 


FIGURE 8.3 The unity feedback control system configuration. 
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trade-off involved. From Equation 8.18, we would like both S(s) and C(s) to be “small” for all s; but this is 
impossible because S(s) + C(s) = I. SVD analysis of the MIMO frequency response can be important in 
quantifying these issues. 


8.5.1.1 Command Following 


Suppose that the reference (command) signal r(t) is sinusoidal, r(t) = 7eJ°!, Then, as long as d(s) =0 
and n(s) = 0, 


e(t) = eel 
where é = S(jw)r. Thus, 


llell2 < OmaxlS(jo)] - |\7ll2 


Now suppose that r(t) is the superposition of more than one sinusoid. Let Q, be the range of frequencies 
at which the input r(t) has its energy. Then, in order to have good command following, we want 


Omax[S(jo)]K 1 Voe€e Q, (8.19) 


Our objective now is to express this prescription for command following in terms of the loop-transfer 
function T(s) = G(s)K(s). From our earlier discussion 


OmaxlS(jo)] = Omax{UI + T(jo)} 1} 
1 
~ Ominll + TGo)] 


This implies that, for good command following, we must have ojin[I + T(jw)] > 1 for all w € Q,. 
However, as we saw earlier, 


OminL + T(jo)] = Ominl T (jo)] —1 


so it is sufficient that, for all w € Q;, Omin[T (jw)] >> 1 for good command following. 


8.5.1.2 Disturbance Rejection 


Suppose that the disturbance signal d(t) is sinusoidal, d(t) = dejt. Then, as long as r(s) = Oand n(s) = 0, 
e(t) = eI! 
where @ = —S(jw)d. Thus, 
lell2 < OmaxlS(jo)] - [Idll2 


Now suppose that d(t) is the superposition of more than one sinusoid. Let Qg be the range of frequencies 
at which the input d(t) has its energy. Then, just as with command following, for good disturbance 
rejection, we want 


OmaxlS(jo)] K1 Vo e Qg (8.20) 


Using the same argument given earlier, this prescription for disturbance rejection makes it sufficient that, 
for all w € Qg, Omin[T(jw)] > 1. 
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8.5.1.3 Relationships to C(s) 


Here we wish to determine in a precise quantitative way the consequences of obtaining command fol- 
lowing and disturbance rejection. As we shall see, a price is paid in constraints on the complementary- 
sensitivity function C(s). 


Theorem 8.2: 


Let Qp = Q, LU Qq. (Here “p” refers to “performance”.) Consider 8 so that0 <8 <1. If 
OmaxlS(jo)] <8 K1 
for all @ € Qp, then no 
1K —— S OminlT(Jo)] 
and 
1-8 < Omin[C(Jo)] < OmaxlC(jo)] < 1+8 
for allo € Qp. 


Thus, in obtaining a performance level of 8, it is necessary that all of the singular values of C( jw) are 
within 8 of 1. In fact, because S(s) + C(s) = I, we must have C( jw) © I. (We shall discuss below why this 
can be a problem.) 


Proof 8.1. We start by using the definition of S(s), 


SmaxlS(jo)] = Omax{(I + T(j@))~'] 
= 1 
= OminU + T(jo)] 
1 
= Gp pala PHA pee N 
1+ OminLT (jo)] 
Using the hypothesis that o;9x[S(jw)] < 8, 
1 
1+ OminlT(jw)] 
which by solving for Omin[ T(j)] yields the first inequality. By cross-multiplying, we obtain the following 


useful expression: 
1 


SminfT joy] ~ 1—8 
Now consider the complementary-sensitivity function C(s). C(s) = [I+ T(s)]7!T(s). By taking the 
inverse of both sides, C~!(s) = T~!(s)[I + T(s)] =I1+ T71(s). Thus 
1 
OminlC(jo)] 


<1 (8.21) 


= OmaxlC'(jw)] = OmaxlI + T~!(jo)] 


which implies 


1 a 
SminlCJo)] <14+0max[T (jo)] 
1 
+ 
OminlT(jo)] 
8 1 
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(Notice that the second inequality follows from Equation 8.21.) Now, 
1-8< OminlC(jo)] = OmaxlC(jo)] 
= OmaxlI — S(jw)] <1+ OmaxlS(jo)] <14+58 


which is the second desired inequality. 


8.5.1.4 Measurement Noise Insensitivity: A Conflict! 
Suppose that the measurement noise n(f) is sinusoidal, n(t) = neJ®!. Then, as long as r(s) = Oandd(s) = 0, 
e(t) = ee" 
where @ = C(jw)d. Thus, 
ell2 < OmaxC(jo)] - llalle 


Now suppose that n(t) is the superposition of more than one sinusoid. Let Q, be the range of frequencies 
at which the input n(t) has its energy. Then, in order to be insensitive to measurement noise, we want 


OmaxlC(jo)] K1 Vw e Qy (8.22) 


Theorem 8.3: 


Let y be such thatO <y « 1. If 
SmaxlC(jw)] < y 
for all w € Qy, then 
OminlT (7)] < OmaxlT(jo)] < i wy<1 


and 
1+1l-y< OminIS(jo)] < Omax(S(jo)] 
forall € Qn. 


Thus, if the complementary-sensitivity function C(jw) has low gain on Q,, then so does the loop- 
transfer function T( jw). This in turn implies that the sensitivity transfer function S(jw) has nearly unity 
gain on Q,. In other words, wherever (in frequency) we are insensitive to measurement noise we are 
necessarily prone to poor command following and disturbance rejection. This is primarily a consequence 
of the fact that C(s) + S(s) =I. 


Proof 8.2. To prove the first relationship we use the fact that C(s) = [I++ T(s)]~!T(s) =[T-(s) +1]! 
(proved using a few algebraic manipulations). This gives 
1 
OminL T—!(jo) +7] 
1 
> 
= Omin[T~!(jo)] +1 
= OmaxlT(jo)] 
~ 1+4+0max[T(jo)] 


Omax[C(jo)] = 


Thus, 


OmaxlT(jo)] SS OmaxlC(jo)] + OmaxlC(Jjwo)]OomaxlT (Go)] 
<yt VOmaxl T(jo)] 
which yields the desired inequality. 
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To prove the second relationship, observe that 


OmaxlS(jo)] ze Ominl[S(jo)] 
= oOmini(I + T(jo))~"] 
me 1 
7 Omaxl + T(jo)] 
Sane ES 
~ OmaxLT(jo)] +1 
by 


IV 


where the last inequality comes from the first relationship proved above. 


8.5.1.5 Design Implications 


Achievable control design specifications must have a wide separation (in frequency) between the sets 
Qy = Q, U Qg and Qy. We cannot obtain good command following and disturbance rejection when we 
have sensors that are noisy on Q,. Figure 8.4 illustrates a problem that is well-posed in terms of these 
constraints. 


Omax S(jw) 


Snax CU w) 


0 < w 
Omax T(jw) 

0< w 

yW(1-y) 


FIGURE 8.4 A well-posed problem: the singular value traces fall within the regions defined by Qp and 8 (for 
performance) and Q,, and y (for noise insensitivity). 
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w(s) z(s) 
-—__————_»> 


P(s) 
u(s) ys) 
es 


FIGURE 8.5 A generalized plant. 


8.5.2 A More General Setting 


Recent control design methodologies are convenient for more generalized problem descriptions. Consider 
the LTI system P shown in Figure 8.5, where u denotes the control vector input to the plant, y represents the 
measurement vector, w is a generalized disturbance vector, and z is a generalized performance vector. 
The assignment of physical variables to w and z here is arbitrary and is left to the control system analyst. 
One illustration is given in Example 8.6. We see that the transfer function matrix for this system can be 
partitioned as follows: 


Pyi(s) Py2(s) 
oleae OI 


From this partition, 


Z(s) = Py1(s)w(s) + Pi2(s)u(s) 
y(s) = Poi (s)w(s) + P22(s)u(s) 


Let K be a feedback controller for the system so that u(s) = K(s)y(s). This feedback interconnection is 
shown in Figure 8.6. It is simple to verify that 


y(s) = [I — P22(s)K(s)]" P21 (s)w(s), 
z(s) = {P11(s) + Py2(s)K(s)[I — P22(s)K(s)] 'P21(s)}w(s), 
= F(P, K)(s)w(s). 


w(s) z(s) 
> > 
P(s) 
u(s) ys) 
>| 
K(s) <____ 


FIGURE 8.6 A feedback interconnection. 
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FIGURE 8.7 Unity feedback example. 


The closed-loop transfer function from w(s) to z(s), denoted F(P, K)(s), is called the (lower) linear 
fractional transformation (of the plant P and K). This type of mathematical object is often very useful in 
describing feedback interconnections. 


Example 8.6: Relationships 


In this example we show how the classical unity-feedback setup can be mapped into the general 
formulation of this section. Consider the block diagram shown in Figure 8.7. With four generalized 
disturbance inputs, a command input r, a sensor noise input n, a system-output disturbance do, and 
a system-input disturbance dj. (These variables are lumped into the generalized disturbance vector 
w.) There are two generalized performance variables, control (effort) u and tracking error e. (These 
are lumped into the generalized performance vector z.) The variables u and y are the control inputs 
and sensor outputs of the generalized plant P. 


8.5.2.1 Frequency Weights 


In control design or analysis it is often necessary to incorporate extra information about the plant 
and its environment into the generalized description P. For example, we may know that system-input 
disturbances are always low-frequency in nature, and/or we may only care about noise rejection at certain 
high frequencies. This kind of information can be incorporated as “frequency weights” augmenting the 
generalized inputs and outputs of the plant P. Such weighting functions Wq(s) and W,(s) are included in 
the block diagram of Figure 8.8. Examples are given below. 


WS) wis) M9), 28), wi L 30 
P(s) 
u(s) I ws) 


FIGURE 8.8 Weighting functions and the generalized plant. 
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With frequency-dependent weightings we can define a new generalized plant P,,. Referring to Figure 8.8 
and using the block partition of P discussed earlier, 


W,(s)Pii(s) Wa(s) Lana 


PYG | Poi(s)Wa(s) Pxa(s) 


With the weighting functions augmenting the plant description, Py is ready (at least conceptually) for 
control design or analysis. If state-space techniques are being employed, the dynamics (i.e., the state vari- 
ables) for Wz and Wp must be included in the state-space representation of Py. This can be accomplished 
by state augmentation. 

Typically, weights Wy on the generalized disturbance vector emphasize the frequency regions of most 
significant disturbance strength. Here w(s) = Wg(s)w1(s). We choose Wg so that a complete description 
of the generalized disturbance is obtained with ||w1(j)|| = 1 for all w. Thus, we may think of Wg 
as an active filter which emphasizes (or de-emphasizes) certain variables in specific frequency domains 
consistent with our understanding of the system’s environment. Reference inputs and system disturbances 
are most often low frequency, so these parts of Wg are usually low-pass filters. On the other hand, certain 
noise inputs are notched (like 60 Hz electrical hum) or high frequency. These parts of Wq(s) should be 
band-pass or high-pass filters, respectively. 

Weights W, on the generalized performance vector emphasize the importance of good performance in 
different frequency regions. For example, we may be very interested in good tracking at low frequencies 
(but not at high frequencies), so that the weighting on this part of W,(s) should be a low-pass filter. On 
the other hand, we may not want to use control energy at high frequencies, so we would choose this part 
of W,(s) as high-pass. The various components of W,(s) must be consistent with one another in gain. 
The relative weights on variables in z should make sense as a whole. 

It is important to note that we must still operate within constraints to mutually achieve various types 
of performance, as was the case with the classical unity-feedback formulation. Because we are no longer 
constrained to that rigid type of feedback interconnection, general results are difficult to state. The 
basic idea, however, remains that there is an essential conflict between noise-rejection and command- 
following/disturbance-rejection. In general there must be a separation in frequency between the regions 
in which the respective types of performance are important. 


8.5.2.2 Weighted Sensitivity 


We consider here the case of only one generalized input variable, d, a system-output disturbance. We are 
also interested in only one performance variable y, the disturbed output of the plant. Specifically, we have 
in mind a weighting function W, which reflects our specifications for y. The feedback configuration is 
shown in Figure 8.9. 


oil w,(s) 2 


v 


e(s) K(s) u(s) G(s) + xs 


FIGURE 8.9 Feedback interconnection for weighted sensitivity. 
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v 


z(s) 
W, (s) > 


e(s) u(s) y(s) 
K(s) G(s) > 


n(s) + At 


FIGURE 8.10 Feedback interconnection for weighted complementary sensitivity. 


It is not hard to see that y(s) = S(s)d(s). Thus, z(s) = Wp(s)S(s)d(s). We refer to W,(s)S(s) as the 
“weighted” sensitivity transfer function for the feedback system. If Omax[Wp(jw) S(j@)] < 1 for all a, 
then, 


OmaxlS(jo)] = OmaxlW, (jo) Wp( jo)S(jo)] 
<OmaxlWp (jo) ]Omaxl Wp(jo)S(jo)] 
< OmaxlW, '(jo)] 
To interpret this, when omax[Wp(jo)S(jo)] < 1 for all «, then the largest singular value of the sensitivity 
transfer function is strictly less than the largest singular value of the inverse of the weighting function. 
8.5.2.3 Weighted Complementary Sensitivity 


We consider here the case of only one generalized input variable, n, a sensor noise input. We are also 
only interested in one performance variable y, the output of the plant. Once again, we have in mind a 
weighting function W, which reflects our specifications for y. The feedback configuration is shown in 
Figure 8.10. 

It is not hard to see that y(s) = C(s)d(s). Thus, z(s) = Wp(s)C(s)d(s). We refer to W>(s)C(s) as 
the “weighted” complementary-sensitivity transfer function for the feedback system. Notice that if 
OmaxlWp(jw)C(jow)] < 1 for all o then, 


OmaxlC(jo)] = Omaxl Wp | (jo) Wp( jo)C(jo)] 
< OmaxlWp | (J0)]Omaxl Wp( jo) C(jo)] 
< OmaxlW, '(jo)] 


To interpret this, when omax[Wp(jo)C(jo)] <1 for all w, then the largest singular value of the 
complementary-sensitivity transfer function is strictly less than the largest singular value of the inverse 
of the weighting function. 
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9.1 Introduction 


In designing feedback control systems, the stability of the resulting closed-loop system is a primary 
objective. Given a finite dimensional, linear time-invariant (FDLTI) model of the plant, G(s), the stability 
of the nominal closed-loop system based on this model, Figure 9.1, can be guaranteed through proper 
design: in the Nyquist plane for single-input, single-output (SISO) systems or by using well-known 
design methodologies such as LQG and Hoo for multi-input, multi-output (MIMO) systems. In any case, 
since the mathematical model is FDLTI, this nominal stability can be analyzed by explicitly calculating the 
closed-loop poles of the system. It is clear, however, that nominal stability is never enough since the model 
is never a true representation of the actual plant. That is, there are always modeling errors or uncertainty. 
As a result, the control engineer must ultimately ensure the stability of the actual closed-loop system, 
Figure 9.2. In other words, the designed controller, K(s), must be robust to the model uncertainty. In this 
article, we address this topic of stability robustness. We present a methodology to analyze the stability of 
the actual closed-loop system under nominal stability to a given model and a certain representation of 
the uncertainty. 
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Ear K(s) {5 G(s) yoy 


Controller Plant model 


FIGURE 9.1 Block diagram of nominal feedback loop. 


The outline of the chapter is as follows. We first establish a representation of the uncertainty on which 
we will base our analysis. We then proceed to derive conditions that guarantee the stability of the actual 
closed-loop system. First, we concentrate on SISO systems where we use the familiar Nyquist stability 
criterion to derive the stability robustness condition. We then derive this same condition using the small 
gain theorem. The purpose of this is to provide a simple extension of our analysis to MIMO systems, 
which we present next. We then interpret the stability robustness conditions and examine their impact 
on attainable closed-loop performance, such as disturbance rejection and command following. Finally, 
we present a discussion on other possible representations of uncertainty and their respective stability 
robustness conditions. Examples are presented throughout the discussion. 


9.2 Representation of Model Uncertainty 


9.2.1 Sources of Uncertainty 


Before we can analyze the stability of a closed-loop system under uncertainty, we must first understand the 
causes of uncertainty in the model so as to find a proper mathematical representation for it. Throughout 
the discussion we will assume that the actual plant, G,(s), is linear time-invariant (LTI) and that we 
have a nominal LTI model, G(s). Although this assumption may seem unreasonable since actual physical 
systems are invariably nonlinear and since a cause of uncertainty is that we model them as LTI systems, 
we need this assumption to obtain simple, practical results. In practice, these results work remarkably 
well for a large class of engineering problems, because many systems are designed to be as close to linear 
time-invariant as possible. 

The sources of modeling errors are both intentional and unintentional. Unintentional model errors 
arise from the underlying complexity of the physical process, the possible lack of laws for dynamic cause 
and effect relations, and the limited opportunity for physical experimentation. Simply put, many physical 
processes are so complex that approximations are inevitable in deriving a mathematical model. On the 
other hand, many modeling errors are intentionally induced. In the interest of reducing the complexity 
and cost of the control design process, the engineer will often neglect “fast” dynamics in an effort to 
reduce the order or the dimension of the state-space representation of the model. For example, one may 


G(s) on 


Controller Actual plant 


FIGURE 9.2 Block diagram of actual feedback loop. 
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neglect “fast” actuator and sensor dynamics, “fast” bending and/or torsional modes, and “small” time 
delays. In addition, the engineer will often use nominal values for the parameters of his model, such as 
time constants and damping ratios, even though he knows that the actual values will be different because 
of environmental and other external effects. 


9.2.2 Types of Uncertainty 


The resulting modeling errors can be separated into two types. The first type is known as parametric 
uncertainty. Parametric uncertainty refers to modeling errors, under the assumption that the actual plant 
is of the same order as the model, where the numerical values of the coefficients to the differential 
equation, which are related to the physical parameters of the system, between the actual plant and the 
model are different. The second type of uncertainty is known as unstructured uncertainty. In this case, 
the modeling errors refer to the difference in the dynamics between the finite dimensional model and the 
unknown and possibly infinite dimensional actual process. 

In this chapter, we limit ourselves to addressing stability robustness with respect to unstructured 
uncertainty. We do this for two main reasons. First, we can capture the parametric errors in terms of 
the more general definition of unstructured uncertainty. Second and more importantly, unstructured 
uncertainty allows us to capture the effects of unmodeled dynamics. From our discussion above, we 
admit that we often purposely neglect “fast” dynamics in an effort to simplify the model. Furthermore, it 
can be argued that all physical processes are inherently distributed systems and that the modeling process 
acts to lump the dynamics of the physical process into a system that can be defined in a finite dimensional 
state-space. Therefore, unmodeled dynamics are always present and need to be accounted for in terms of 
stability robustness. 


9.2.3 Multiplicative Representation of Unstructured Uncertainty 


What we need now is a mathematical representation of unstructured uncertainty. The difficulty is that 
since the actual plant is never exactly known, we cannot hope to model the uncertainty to obtain this 
representation, for otherwise, it would not be uncertain. On the other hand, in practice, the engineer 
is never totally ignorant of the nature and magnitude of the modeling error. For example, from our 
arguments above, it is clear that unstructured uncertainty cannot be captured by a state-space represen- 
tation, since the order of the actual plant is unknown, and thus we are forced to find a representation in 
terms of input-output relationships. In addition, if we choose to neglect “fast” dynamics, then we would 
expect that the magnitude of the uncertainty will be large at high frequencies in the frequency domain. In 
any case, the key here is to define a representation that employs the minimal information regarding the 
modeling errors and that is, in turn, sufficient to address stability robustness. 


9.2.3.1 Set Membership Representation for Uncertainty 


In this article we adopt a set membership representation of unstructured uncertainty. The idea is to define 
a bounded set of transfer function matrices, G, which contains G,(s). Therefore, if G is properly defined 
such that we can show stability for all elements of G, then we would have shown stability robustness. 
Towards this end, we define G as 


G = {G(s) | G(s) = (I+ w(s)A(s)) G(s), | ACjo) la. < U (9.1) 


where 


1. w(s) isa fixed, proper, and strictly stable scalar transfer function 
2. A(s) isa strictly stable transfer function matrix (TFM) 
3. No unstable or imaginary axis poles of G(s) are cancelled in forming G(s) 
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This is known as the multiplicative representation for unstructured uncertainty. Since it is clear that 
the nominal model G(s) is contained in G, we view G as a set of TFMs perturbed from G(s) that covers 
the actual plant. Our requirements that w(s)A(s) is strictly stable and that there are no unstable or 
imaginary axis pole-zero cancellations mean that the unstable and imaginary axis poles of our model and 
any G(s) € G coincide. This assumes that the modeling effort is at least adequate enough to capture the 
unstable dynamics of G,(s). 

In Equation 9.1, the term w(s)A(s) is known as the multiplicative error. We note that since the Hoo 
norm of A(jw) varies between 0 and 1 and since the phase and direction of A(jw) are allowed to vary 
arbitrarily, the multiplicative error for any G(s) € G is contained in a bounded hypersphere of radius 
|w(ja)| at each frequency. Therefore, our representation of unstructured uncertainty is one in which we 
admit total lack of knowledge of the phase and direction of the actual plant with respect to the model, but 
that we have a bound on the magnitude of this multiplicative error. This magnitude-bound information 
is frequency dependent and is reflected in the fixed transfer function w(s), which we will refer to as the 
weight. We note that we could have used a TFM W(s) instead of a scalar w(s) in our representation. 
However, in that case, the multiplicative error will reflect directional information of W(s). Since we have 
presumed total lack of knowledge regarding directional information of the actual plant relative to the 
model, it is common in practice to choose W(s) to be scalar. 


9.2.3.2 SISO Interpretations for G 


To get a better feel for G and our representation for unstructured uncertainty, we first specialize to the 
SISO case in which our definition for G gives 


&(s) = (1+ w(s)A(s))g(s) (9.2) 


where 
|ACjo) llaz,, = sup |ACGjw)| < 1 (9.3) 


Since the phase of A(jw) is allowed to vary arbitrarily and its magnitude varies from 0 to 1 at all frequen- 
cies, the set G is the set of transfer functions whose magnitude bode plot lies in an envelope surrounding 
the magnitude plot of g(s), as shown in Figure 9.3. Therefore, the size of the unstructured uncertainty 


A 


(1+]w(jo)|)|gGo)| 


lgGo)| 


(1-|w(jo)|)|gGo)| 


Magnitude 


Envelope for actual plant 


Frequency 


FIGURE 9.3 Bode plot interpretation of multiplicative uncertainty. 
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is represented by the size of this envelope. From the figure, the upper edge of the envelope corresponds 
to the plot of (1+ |w(ja)|)|g(ja)| while the lower edge corresponds to the plot of (1 — |w(jw)|)|g(jo)|- 
Therefore, |w(jw)| is seen as a frequency-dependent magnitude bound on the uncertainty. As mentioned 
beforehand, the size of the unstructured uncertainty typically increases with increasing frequency. There- 
fore, we would typically expect the size of the envelope containing G to increase with increasing frequency 
and also |w(jw)| to increase with increasing frequency, as shown in Figure 9.3. Furthermore, we want 
to stress that since the phase of A( jw) is allowed to vary arbitrarily, the phase difference between any 
gZ(jw) € G and g(jw) can be arbitrarily large at any frequency. 

For another interpretation, we can look at the multiplicative error w(s)A(s) in the SISO case. Solving 
for w(s)A(s) in Equation 9.2 gives 


a = w(s) A(s) (9.4) 


which shows that w(s)A(s) is the normalized error in the transfer function of the perturbed system with 
respect to the nominal model. Using Equation 9.3 and noting that everything is scalar, we take magnitudes 
on both sides of (Equation 9.4) for s = jw to get 


Ig(jo) — g(Jo)| < |wGo)||AG@)||gGo)| < lwGo)llgGio)| Vo (9.5) 


As shown in Figure 9.4, for each w, this inequality describes a closed disk in the complex plane of radius 
|w(j)||g(jw)| centered at g( jw) which contains g( jw). Since Equation 9.5 is valid for any g(s) € G, the 
set G is contained in that closed disk for each w. In this interpretation, the unstructured uncertainty is 
represented by the closed disk, and therefore, we see that the direction and phase of the uncertainty is left 
arbitrary. However, we note that the radius of the closed disk does not necessarily increase with increasing 
frequency because it depends also on |g(jw)|, which typically decreases with increasing frequency at high 
frequencies due to roll-off. 


9.2.3.3 Choosing w(s) 


From our discussion, it is clear that our representation of the uncertainty only requires a nominal 
model, G(s), and a scalar weight, w(s), which reflects our knowledge on the magnitude bound of the 
uncertainty. The next logical question is how to choose w(s) in the modeling process. From the definition 


A Im 


Ig(ja)| 


|w(jo)||gGo)| 


FIGURE 9.4 Interpretation of multiplicative uncertainty in the complex plane. 
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in Equation 9.1, we know that we must choose w(s) so that the actual plant is contained in G. In the course 
of modeling, whether through experimentation, such as frequency response, and/or model reduction, we 
will arrive at a set of transfer function matrices. It is assumed that our modeling effort is thorough enough 
to adequately capture the actual process so that this set will cover the TFM of the actual plant. From 
this set we will choose a nominal model, G(s). We assume that G(s) is square, same number of inputs as 
outputs, and nonsingular along the jw-axis in the s-plane. With this assumption, we can calculate, at each 
frequency, the multiplicative error for each TFM G;(s) in our set using 

w(jw) A(jo) = Gi(jo)G7 (jo) — 1 (9.6) 

Taking maximum singular values on both sides of the above equation gives 
Smaxl( jo) A(jo)] = OmaxlGi(jo)G~' (jo) — 1] (9.7) 

Since || A(jw) ||47,, < 1 and w( jo) is a scalar, it is clear that we must choose 
|w(jo)| = OmaxlGi(jo)G (jo) -—I] Va >0 (9.8) 


to include the TFM G;(s) in G. Therefore, we must choose a stable, proper w(s) such that 


|w(jw)| > max OmaxlGi(jo)G~ (jw) —I] Vo >0 (9.9) 


to ensure that we include all G;(s) and, thus, the actual plant in G. This process is illustrated in the 
following example. 


Example 9.1: Integrator with Time Delay 
Consider the set of SISO plants 
1 
gr(s) = — exp **, O0<1<02 (9.10) 
S 


which is the result of our modeling process on the actual plant. Since we cannot fully incorporate 
the delay in a state-space model, we choose to ignore it. As a result, our model of the plant is 


g(s) = — (9.11) 


To choose the appropriate w(s) to cover all gx(s) in G, we have to satisfy Equation 9.9, which in the 
SISO case is 


|w(jo)| > max a | Vo>0 (9.12) 
(60) 
Therefore, we need to choose w(s) such that 
\w(joo)| > max eH" — | ¥as0 (9.13) 


Using e JT — cos(wt) —jsin(wt) and a few trigonometry identities, the above inequality can be 
simplified as 
|w(jo)] = max |2 sin (+)| Vo>0 (9.14) 


Stability Robustness to Unstructured Uncertainty for Linear Time Invariant Systems 9-7 


20 : 
Iw(jo] : 

10 : ay a 
' 

h 

sf) l 


I 
un 
o 


Magnitude (dB) 


20) frrenegndbaned 


: Maximum multiplicative 
: error for g(s) = 1/s 


30 bivughndonzs 


40 i 3 - 2 : 
1077 10° 10! 10? 103 
Frequency (rad/s) 


FIGURE 9.5 __w/(s) for time delay uncertainty. 


A simple w(s) that will satisfy Equation 9.14 is 


0.215 


w(s) = ————_ 
0.05s + 1 


(9.15) 


This is shown in Figure 9.5, where |w(jw)| and 2 |sin(*)| are plotted together on a magnitude bode 
plot for t = 0.2, which is the worst-case value. We note that w(s) is proper and strictly stable as required. 

Now, let us suppose that we seek a better model by approximating the delay with a first-order Pade 
approximation 


7 St 
etry 2 (9.16) 
1+— 
2 
Our model becomes 
0.1s 
7 ae 
g(s) = 2. was (9.17) 
1+ — 
2 


where we approximate the delay at its midpoint value of 0.1 s. To choose w(s) in this case, we again need 
to satisfy Equation 9.12 which becomes 


e JT (jo + 20) 


|w(jo)| > max Go —20) Vo>0 (9.18) 
A simple w(s) that will satisfy Equation 9.18 is 
0.11s 
= —_ 9.19 
we) 0.025s+ 1 ( ) 


This is shown in Figure 9.6. We again note that w(s) is proper and strictly stable as required. 
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FIGURE 9.6 _w/(s) for time delay uncertainty with Pade approximation. 


Comparing the magnitudes of w(s) for the two models in Figure 9.7, we note that, for the model with 
the Pade approximation, |w(jo)| is less than that for the original model for w < 100 rad/s. Since |w(ja)| 
is the magnitude bound on the uncertainty, Figure 9.7 shows that the uncertainty for the model with the 
Pade approximation is smaller than that for the original model in this frequency range. Physically, this is 
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FIGURE 9.7 Comparing w(s) for the two models. 
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because the model with the Pade approximation is a better model of the actual plant for w < 100 rad/s. 
Asa result, its uncertainty is smaller. 

We note that our choice of representing the uncertainty as being bounded in magnitude but arbitrary 
in phase and direction is, in general, an overbound on the set of TFMs that we obtained from the modeling 
process. That is, the set of TFMs from the modeling process may only be a small subset of G. The benefit of 
using this uncertainty representation is that it allows for simple analysis of the stability robustness problem 
using minimal information regarding the modeling errors. However, the cost of such a representation 
is that the stability robustness results obtained may be conservative. This is because these results are 
obtained by showing stability with respect to the larger set G instead of the smaller set from our modeling 
process. Another way of looking at it is that the resulting set of stabilizing controllers for the larger set G 
will be smaller. As a result, the achievable performance may be worse. This conservatism will be discussed 
throughout as we develop our analysis for stability robustness, and the impact of stability robustness and 
this conservatism on closed-loop performance will be discussed in Section 9.4. 


9.2.3.4 Reflecting Modeling Errors to the Input of the Plant 


Finally, we note that in our representation of multiplicative unstructured uncertainty, we choose to 
lump the uncertainty at the plant output. In other words, we assume that the actual plant is of the form 
(I+ w(s)A(s))G(s) where the uncertainty, (I + w(s)A(s)) multiplying the model is at the plant output. 
To be more precise notationally, we should rewrite our definition for G as 


G = {G(s) | G(s) = (I+ wo(s)A(s))GGs), |AG@) Ila. < DB (9.20) 


«>» 


where we have added the subscript “o” to w(s) to distinguish it as the weight corresponding to uncertainty 
at the plant output. Alternatively, we can instead choose to lump the uncertainty at the plant input. In this 
case, we assume that the actual plant is of the form G(s)(J + w;(s)A(s)) where the uncertainty multiplying 
the model is at the plant input. As a result, the definition for G in this case is 


G ={G(s) | G(s) = G(s)\(I + wi(s)A(s)), | AGi@) Ila, < 1) (9.21) 


where w;(s) is the weight corresponding to the uncertainty at the plant input. Comparing the two rep- 
resentations reveals that, in general, the two sets defined by Equations 9.20 and 9.21 are not the same 
because matrix multiplication does not commute. Since A(s) is constrained similarly and we must choose 
the weight so that G covers the actual plant in both cases, we note that, in general, w,(s) is not the same 
as wj(s). 

Of course, we do not physically lump the modeling errors to the plant input or output. Instead, in the 
course of modeling, we lump the model so that the modeling errors are reflected either to the input of the 
model or to the output. To be sure, modeling errors associated with the plant actuators and sensors, such 
as neglected actuator and sensor dynamics, are reflected more naturally to the model’s input and output, 
respectively. However, modeling errors associated with internal plant dynamics, such as neglected flexible 
modes, are not naturally reflected to either the model’s input or output. In this case, we have a choice as 
to where we wish to reflect these errors. As a final note, for the SISO case, the two representations are 
equivalent since scalar multiplication does commute. As a result, we can choose wo(s) = wj(s) = w(s) so 
that it does not make a difference where we reflect our modeling errors. 


9.3 Conditions for Stability Robustness 


Having established a multiplicative representation for unstructured uncertainty, we proceed to use it 
to analyze the stability robustness problem. As mentioned beforehand, since the actual modeling error 
and, thus, the actual plant is never known, we cannot hope to simply evaluate the stability of the actual 
closed-loop system by using the Nyquist stability criterion or by calculating closed-loop poles. Instead, we 
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must rely on our representation of the uncertainty to arrive at conditions or tests in the frequency domain 
that guarantee stability robustness. Throughout the discussion, we assume that we have a controller, K(s), 
that gives us nominal stability for the feedback loop in Figure 9.1 and that we use this controller in the 
actual feedback loop, Figure 9.2. In addition, since we are interested in internal stability of the feedback 
loop, we assume throughout that for SISO systems, the transfer function k(s)g(s) is stabilizable and 
detectable; that is, there are no unstable pole-zero cancellations in forming k(s)g(s). For MIMO systems, 
the corresponding conditions are that both K(s)G(s) and G(s)K(s) are stabilizable and detectable. 


9.3.1 Stability Robustness for SISO Systems 


In this section we analyze the stability robustness of SISO feedback systems to multiplicative unstructured 
uncertainty. We first derive a sufficient condition for stability robustness using the familiar Nyquist 
stability criterion. We then derive the same condition using another method: the small gain theorem. The 
goal here is not only to show that one can arrive at the same answer but also to present the small gain 
approach to analyzing stability robustness, which can be easily extended to MIMO systems. Finally, we 
compare our notion of stability robustness to more traditional notions of robustness such as gain and 
phase margins. 


9.3.1.1 Stability Robustness Using the Nyquist Stability Criterion 


We begin the analysis of stability robustness to unstructured uncertainty for SISO systems with the Nyquist 
stability criterion. We recall from classical control theory that the Nyquist stability criterion is a graphical 
representation of the relationship between the number of unstable poles of an open-loop transfer function, 
I(s), and the unstable zeros of the return difference transfer function, 1+ I(s). Since the zeros of 1 + I(s) 
correspond to the closed-loop poles of /(s) under negative unity feedback, the Nyquist stability criterion 
is used to relate the number of unstable poles of /(s) to the stability of the resulting closed-loop system. 
Specifically, the Nyquist stability criterion states that the corresponding closed-loop system is stable if and 
only if the number of positive clockwise encirclements (or negative counterclockwise encirclements) of 
the point (—1, 0) in the complex plane by the Nyquist plot of /(s) is equal to —P, where P is the number of 
unstable poles of I(s). Here, the Nyquist plot is simply the plot in the complex plane of /(s) evaluated along 
the closed Nyquist contour D,, which is defined in the usual way with counterclockwise indentations 
around the imaginary axis poles of /(s) so that they are excluded from the interior of D,. Notationally, we 
express the Nyquist stability criterion as 


(-1, [(s), D;) = —P (9.22) 


In our analysis we are interested in the stability of the two feedback loops given in Figures 9.1 and 9.2. 
For the nominal feedback loop, we define the nominal loop transfer function as 


I(s) = g(s)k(s) (9.23) 
Since we assume that the nominal closed loop is stable, we have that 
X(—1, g(s)k(s), Dy) = —P (9.24) 


where P is the number of unstable poles of g(s)k(s). For the actual feedback loop, we define the actual 
loop transfer function as 


Ia(s) = ga(s)k(s) = (1 + w(s)A(s))g(s)k(s) (9.25) 


where the second equality holds for some ||A(jo)||z,, <1 since we assumed that g,(s) € G in our 
representation of the unstructured uncertainty. In addition, since we assumed that the unstable poles of 
g(s) and g,(s) coincide and since the same k(s) appears in both loop transfer functions, the number of 


unstable poles of g,(s)k(s) and thus Jg(s) is also P. Similarly, since we assumed that the imaginary axis 
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poles of g(s) and g,(s) coincide, the same Nyquist contour D, can be used to evaluate the Nyquist plot of 
I,(s). Therefore, by the Nyquist stability criterion, the actual closed-loop system is stable if and only if 


N(—1, ga(s)k(s), Dy) = 8(—1, (1 + w(s)A(s))g(s)k(s), Dy) 
=-—P (9.26) 


Since we do not know g,(s), we can never hope to use Equation 9.26 to evaluate or show the stability 
of the actual system. However, we note that Equation 9.26 implies that the actual closed-loop system is 
stable if and only if the number of counterclockwise encirclements of the critical point, (—1,0), is the 
same for the Nyquist plot of /;(s) as that for /(s). Therefore, if we can show that for the actual loop transfer 
function, we do not change the number of counterclockwise encirclements from that of the nominal 
loop transfer function, then we can guarantee that the actual closed-loop system is stable. The idea is to 
utilize our set membership representation of the uncertainty to ensure that it is impossible to change the 
number of encirclements of the critical point for any loop transfer function g(s)k(s) with g(s) € G. This 
will guarantee that the actual closed-loop system is stable, since our representation is such that g,(s) € G. 
This is the type of sufficient condition for stability robustness that we seek. 

To obtain this condition, we need a relationship between the Nyquist plots of /(s) and J,(s) using our 
representation of the unstructured uncertainty. To start, we separate the Nyquist plot into three parts 
corresponding to the following three parts of the Nyqust contour D;: 


1. The nonnegative jw axis 
2. The negative jw axis 
3. The part of D, that encircles the right half s-plane where |s| + oo 


For the first part, we note from Equations 9.23 and 9.25 that J,(s) can be expressed as 
la(s) = (1+ w(s) A(s))U(s) (9.27) 


for some ||A||7,, < 1. Therefore, the multiplicative uncertainty representation developed earlier holds 
equally for I(s) = g(s)k(s) as for g(s). Extending the SISO interpretation of this uncertainty representation 
in Figure 9.4 through Figure 9.8, we note that for any nonnegative w, the Nyquist plot of J;(s) is a point 
contained in the disk of radius |w(jw)||I(jw)| centered at the point [(jw), which is the Nyquist plot 
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FIGURE 9.8 Multiplicative uncertainty in the Nyquist plot. 
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of I(s) at that w. Figure 9.8, then, gives a relationship between the Nyquist plot of /(s) and [,(s) at a 
particular frequency for the part of the Nyquist contour along the positive jw axis. For the second part 
of the Nyquist contour, we note that the Nyquist plot is simply a mirror image, with respect to the real 
axis, of the Nyquist plot for the first part, and thus this relationship will be identical. For the third part of 
the Nyquist contour, we note that since the model g(s) represents a physical system, it should be strictly 
proper, and therefore, |g(s)| — 0 as |s| + oo. In addition, the same holds for gq(s) since it is a physical 
system. Since the controller k(s) is proper, it follows that the both the Nyquist plots for /(s) and J,(s) are 
at the origin for the part of the Nyquist contour that encircles the right half s-plane. 

For stability robustness, we only need to consider the relationship between /(s) and J,(s) for the first 
part of the Nyquist contour. This is because for the second part the relationship is identical and thus the 
conclusions for stability robustness will be identical. Finally, for the third part, since the Nyquist plots 
for both /(s) and J,(s) are at the origin, they are identical and cannot impact stability robustness. As a 
result, we only need to consider the relationship between /(s) and /,(s) that is presented in Figure 9.8. We 
illustrate this relationship along a typical Nyquist plot of the nominal /(s), for w > 0, in Figure 9.9 where, 
for clarity, we only illustrate the uncertainty disk at a few frequencies. Here, we note that the radius of the 
uncertainty disk changes as a function of frequency since both w( jw) and I( jw) varies with frequency. 

From Figure 9.9, we note that it is impossible for the Nyquist plot of J,(s) to change the number of 
encirclements of the critical point (—1, 0) if the disks representing the uncertainty in /(s) do not intersect 
the critical point for all w > 0. To show this, let us prove the contrapositive. That is, if the Nyquist plot 
of I,(s) does change the number of encirclements of the critical point, then by continuity there exists a 
frequency w* where the line connecting /(jw*) and J,(jw*) must intersect the critical point. Since the 
uncertainty disk at that frequency is convex, this uncertainty disk must also intersect the critical point. 
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FIGURE 9.9 Stability robustness using the Nyquist stability criterion. 
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Graphically, then, we can guarantee stability robustness if the distance between the critical point and the 
Nyquist plot of I(s) is strictly greater than the radius of the uncertainty disk for all w > 0. From Figure 9.8, 
the distance between the critical point and the Nyquist plot of the nominal loop transfer function, [(s), 
is simply |1 + /1(jw)|, which is the magnitude of the return difference. Therefore, a condition for stability 
robustness is 


Iw(jo)|[I(jo)| < |1+1Go)| Vo 2o (9.28) 


or equivalently, 


|w(jo)llgiw)k(j@)| < [1+ g(jo)k(jo)| Vo > 0 (9.29) 


We note that although the Nyquist stability criterion is both a necessary and sufficient condition for 
stability, the above condition for stability robustness is clearly only sufficient. That is, even if we violate 
this condition at a particular frequency or at a range of frequencies, the Nyquist plot of the actual system 
may not change the number of encirclements of the critical point and thus the actual closed-loop system 
may be stable. This is illustrated in Figure 9.10, where we note that if the Nyquist plot of /,(s) follows /g2(s), 
then the actual system is stable. The key here is that since we do not know J,(s), we cannot say whether 
or not the actual system is stable when the condition is violated. Therefore, to guarantee actual stability, 
we need to ensure a safe distance or margin, which may not be necessary, between the nominal Nyquist 
plot and the critical point. This margin is in terms of the uncertainty disk. We see that this conservatism 
stems from the fact that we admit total lack of knowledge concerning the phase of the actual plant, which 
led to the representation of the uncertainty as a disk in the complex plane. 
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FIGURE 9.10 Sufficiency of the stability robustness condition. 
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FIGURE 9.11 Standard feedback form. 


9.3.1.2 Stability Robustness Using Small Gain Theorem 


We now seek to derive an equivalent condition for stability robustness for SISO systems using the 
small gain theorem, see for example Dahleh and Diaz-Bobillo [1]. The goal here is to introduce another 
methodology whereby conditions for stability robustness can be derived. As we will see in the sequel, 
this methodology can be easily extended to MIMO systems. We begin with a statement of the small gain 
theorem, specialized to LTI systems, which addresses the stability of a closed-loop system in the standard 
feedback form given in Figure 9.11. 


Theorem 9.1: Small Gain Theorem 


Under the assumption that G,(s) and G(s) are stable in the feedback system in Figure 9.11, the closed-loop 
transfer function matrix from (uy, U2) to (e1, €2) is stable if the small gain condition 


Gi (J@)[I 41.0 IIG2Go)llH.. <1 (9.30) 
is satisfied. 


Proof. We first show that the sensitivity transfer function matrix, S(s) = (I — G,(s)G,(s))~! is stable. 
For this, we need to show that if the small gain condition Equation 9.30 is satisfied, then S(s) = (I — 
G1(s)Go(s))~! is analytic in the closed right-half s-plane. An equivalent statement is that the return 
difference, D(s) = I — Gi(s)G2(s) is nonsingular for all s in the closed right-half plane. 


For arbitrary input u and for all complex s, 


|| D(s)ul] = || 1 — Gi (s)Ga(s))ul] = |]u — Gy (s)Ga(s)ul| (9.31) 
where || - || represents the standard Euclidean norm. From the triangle inequality, 
\|u — Gy (s)G2(s)ul] = |Iul] — Gi (s)G2(s)ulh (9.32) 


from the definition of the maximum singular value, 
|| Gi (s)Ga(s)ul| < OmaxlGi(s)Ga(s)]||ull (9.33) 
and from the submultiplicative property of induced norms, 


Omaxl Gi (S)G2(s)] < Omaxl Gi (s)]omaxl Go(s)] (9.34) 
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Substituting Equations 9.32 through 9.34 into Equation 9.31 gives for all complex s, 


|| D(s)u|| = ||u — Gy (s)G2(s)ul 
= [lull — Gi (s)G2(s)ull 
= [lull — omaxlGi (s)G2(s)]II ull 
= lull — oOmaxlGi (s)]omaxlG2(s)]II ull 
= (1 — oOmax[Gi (s)]omaxl Ga(s))) lull (9.35) 


Since G; and Gy are stable, they are analytic in the closed right-half plane. Therefore, by the maximum 
modulus theorem [2], 


OmaxlG1(s)] < sup Omaxl Gi (jw)] = IGiGjo) lle. 


(9.36) 
OmaxlG2(s)] < sup Omax[ G2(jw)] _ GoJo) lle. 
for all s in the closed right-half plane. Substituting Equation 9.36 into Equation 9.35 gives 
|D(s)ull = 1. = [Gi Go) 740 G2) Il 41.0) lull (9.37) 


for all s in the closed right-half plane. From the small gain condition, there exists an € > 0 such that 
G1 (J)[I 4450 IIG2Go)lln.. <1—€ (9.38) 
Therefore, for all s in the closed right-half plane, 
|| D(s)ul| > €llul] > 0 (9.39) 


for any arbitrary u, which implies that D(s) is nonsingular in the closed right-half s-plane. 
From a similar argument, we can show that (I — G2(s)Gj(s)) is also stable. Therefore, the transfer 
function matrix relating (11, uz) to (e1, e2), which is given by 


(I= G2(s)Gi(s))"! Oe hes 


(I — Gy (s)Ga(s)~'Gy(s) (I — Gy(s)Ga(s)) 


is stable. 
We note that the small gain theorem is only a sufficient condition for stability. For example, in the 
SISO case, the small gain condition (Equation 9.30) can be expressed as 


Igi(jo)g2(jo)| <1 Vo >0 (9.41) 


which implies that the Nyquist plot of the loop transfer function gj (s)g2(s) lies strictly inside the unit circle 
centered at the origin, as shown in Figure 9.12. This is sufficient for stability because gj(s) and g2(s) are 
stable; however, we note that it is clearly not necessary. In addition, we note that the small gain theorem 
applies equally well to MIMO systems, since the proof was actually done for the MIMO case. In fact, the 
general form of the small gain theorem applies to nonlinear, time-varying operators over any normed 
signal space. For a treatment of the general small gain theorem and its proof, the reader is referred to [3]. 
For stability robustness, we are, as before, interested in the stability of the two feedback loops given 
in Figures 9.1 and 9.2. From our representation of unstructured uncertainty, we can express the actual 

plant as 
&a(s) = (1+ w(s)A(s))g(s) (9.42) 


for some || A(ja)||71,, < 1. Therefore, the actual feedback loop can also be represented by the block 
diagram in Figure 9.13. In the figure, we note that we choose, merely by convention, to reflect the 
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|g, G@)go(j@)| <1 


FIGURE 9.12 Sufficiency of SISO small gain theorem. 


unstructured uncertainty to the output of the plant instead of to the input since for the SISO case the two 
are equivalent. In addition, we note that Figure 9.13 can also be interpreted as being the feedback loop for 
all perturbed plants, ¢(s), belonging to the set G. The key for stability robustness, then, is to show stability 
for this feedback loop using the small gain theorem for all || A( jw) ||77,, < 1 and, therefore, for all g(s) € G. 

To apply the small gain theorem, we first reduce the feedback loop in Figure 9.13 to the standard 
feedback form in Figure 9.11. To do this, we isolate A(s) and calculate the transfer function from the 
output of A, v, to its input, z. From Figure 9.13, 


z(s) = —w(s)g(s)k(s)y(s) 
y(s) = v(s) — g(s)k(s)y(s) (9.43) 


a 
Cw K(s) a als) x 
i 


Actual plant g,(s) 


FIGURE 9.13 Actual feedback loop with uncertainty representation. 
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—f » -— 


—w(s)g(s)k(s) 
1 + g(s)k(s) 


FIGURE 9.14 Actual feedback loop in standard form. 


As a result, the transfer function seen by A is given by m(s) where 


_ —w(s)g(s)k(s) 


m(s) = T+ 2(s)K(s) (9.44) 


and the reduced block diagram is given in Figure 9.14. We note that m(s) is simply the product of the 
complementary sensitivity transfer function for the nominal feedback loop and the weight w(s). Since we 
assumed that the nominal closed loop is stable and w(s) is stable, m(s) is stable. Furthermore, since we 
also assumed that A(s) is stable, the assumption for the small gain theorem is satisfied for the closed-loop 
system in Figure 9.14. Applying the small gain theorem, this closed-loop system is stable if the small gain 
condition 


—w(jo)gjw)k(jo) | AC ; bas 
peruano ae 

is satisfied. Since | A(jo) | Hoo = bean equivalent condition for stability to all | A(jo) | Hoo < Lis 
w(jo)g(jo)k jo) | : 9.46 
| 1+ gGa)kGo) Noy, ~ veo) 


which is a sufficient condition for the stability of the feedback loop in Figure 9.13 for all g(s) € G. Since 
&a(s) € G, this is a sufficient condition for stability robustness. 

We now proceed to show that the stability robustness condition in Equation 9.46 is equivalent to that 
derived using the Nyquist stability criterion. From the definition of the Ho. norm, an equivalent condition 
to Equation 9.46 is given by 

w(jo)gjo)k(jo) 
1+ g(jayk( joo) 


Since everything is scalar, this condition is equivalent to 


<1 Vw>0 (9.47) 


|w(jo)llgiw)k(j@)| < [1+ g(jo)k(jo)| Vo > 0 (9.48) 


which is exactly the condition for stability robustness derived using the Nyquist stability criterion. This 
equivalence is not due to the equivalence of the Nyquist stability criterion and the small gain theorem, 
since the former is a necessary and sufficient condition for stability while the latter is only sufficient. 
Rather, this equivalence is due to our particular approach in applying the small gain theorem and to our 
representation of the uncertainty. What this equivalence gives us is an alternative to the more familiar 
Nyquist stability criterion in analyzing the stability robustness problem. Unlike the Nyquist stability 
criterion, the small gain theorem applies equally well to MIMO systems. Therefore, our analysis is easily 
extended to the MIMO case, as we will do in Section 9.3.2. 
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FIGURE 9.15 Insufficiency of gain and phase margins. 


9.3.1.3 Comparison to Gain and Phase Margins 


In closing this section on stability robustness for SISO systems, we would like to compare our notion 
of robustness to more traditional notions such as gain and phase margins. As we recall, gain margin is 
the amount of additional gain that the SISO open-loop transfer function can withstand before the closed 
loop goes unstable, and phase margin is the amount of additional phase shift or pure delay that the loop 
transfer function can withstand before the closed loop goes unstable. To be sure, gain and phase margins 
are measures of robustness for SISO systems, but they are, in general, insufficient in guaranteeing stability 
in the face of dynamic uncertainty such as those due to unmodeled dynamics. This is because gain and 
phase margins only deal with uncertainty in terms of pure gain variations or pure phase variations but 
not a combination of both. That is, the open loop can exhibit large gain and phase margins and yet be 
close to instability as shown in the Nyquist plot in Figure 9.15. In the figure, we note that for frequencies 
between w and wz the Nyquist plot is close to the critical point so that a combination of gain and phase 
variation along these frequencies such as that in the perturbed Nyquist plot will destabilize the closed 
loop. This combination of gain and phase variations can be the result of unmodeled dynamics. In such a 
case, gain and phase margins will give a false sense of stability robustness. In contrast, we could get a true 
sense of stability robustness by explicitly accounting for the dynamic uncertainty in terms of unstructured 
uncertainty. 

In addition, gain and phase margins are largely SISO measures of stability robustness since they are 
inadequate in capturing the cross coupling between inputs and outputs of the dynamics of MIMO systems. 
For MIMO systems, we usually think of gain and phase margins as being independent gain and phase 
variations that are allowed at each input channel. These variations clearly cannot cover the combined 
gain, phase, and directional variations due to MIMO dynamic uncertainty. As a result, the utility of our 
notion of stability robustness over traditional gain and phase margin concepts becomes even more clear 
in the MIMO case. 


9.3.2 Stability Robustness for MIMO Systems 


In this section, we analyze the stability robustness of MIMO feedback systems to multiplicative unstruc- 
tured uncertainty. As mentioned beforehand, we will use the small gain theorem since it offers a natural 
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extension from the SISO case to the MIMO case. As shown in the SISO case, the general procedure for 
analyzing stability robustness using the small gain theorem is as follows: 


1. Start with the block diagram of the actual feedback loop with the actual plant represented by the 
nominal model perturbed by the uncertainty. Note that this block diagram also represents the 
feedback loop for all perturbed plants belonging to the set G, which contains the actual plant. 

2. Reduce the feedback loop to the standard feedback form by isolating the A(s) block and calculating 
the TFM from the output of A to the input of A. Denote this TFM as M(s). 

3. Apply the small gain theorem. In particular, since | A(jo) | Hoey the small gain theorem guar- 
antees stability for the feedback loop for all perturbed plants in the set G and therefore guarantees 
robust stability if || M( jo) lees: <1, 


We will follow this procedure in our analysis of MIMO stability robustness. For MIMO systems, as shown 
in Section 9.2.3, there is a difference between reflecting the modeling errors to the input and the output of 
the plant. As a consequence, we separate the two cases and derive a different stability robustness condition 
for each case. In the end, we will relate these two stability robustness tests and discuss their differences. 


9.3.2.1 Uncertainty at Plant Output 


We start with the case where the modeling errors are reflected to the plant output. In this case, the actual 
plant is of the form 


Ga(s) = UI + wo(s)A(s))G(s) (9.49) 
for some | A(jo) | 41. < 1. Following step 1 of the procedure, the block diagram of the actual feedback 


loop given this representation of the uncertainty is depicted in Figure 9.16a. For step 2, we calculate the 
TEM from the output of A, v, to its input, z. From Figure 9.16a, 


2(s) = —Wo(s)G(s)K(s)y(s) 
y(s) = v(s) — G(s)K(s)y(s) (9.50) 


Asa result, M(s), the TFM seen by A(s), is 
M(s) = —w,(s)G(s)K(s)\U + G(s)K(s))7! (9.51) 


and the reduced block diagram is given in Figure 9.16b. We note that M(s) is simply the product of the 
complementary sensitivity TFM, C(s) = G(s)K(s)(I + G(s)K(s))~1, for the nominal feedback loop and 
the scalar weight wo(s). Since we assumed that the nominal closed loop and w,(s) are both stable, M(s) is 
stable. Furthermore, since we also assumed that A(s) is stable, the assumption for the small gain theorem 
is satisfied for the closed loop system in Figure 9.16b. For step 3, we apply the small gain theorem, which 
gives 

|| wo( ja)G(ja)K (jw)(I + G(jo)K(ja))! ee <1 (9.52) 


as a sufficient condition for stability robustness. Using the definition of the Ho, norm, an equivalent 
condition for stability robustness is 


OmaxlWo( jw) G( jw)K (jw)(I + G(jw)K(jo))']} <1 Vo >o (9.53) 


Since wo(s) is scalar, another sufficient condition for stability robustness to multiplicative uncertainty at 
the plant output is 


Omaxl G(j@)K(jo)(I + G(jo)K(jo))'] < (9.54) 


|wo(jo)| 
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—w,(s)G(s)K(s)(1+G(s)K(s))7? 


FIGURE 9.16 Stability robustness for multiplicative uncertainty at plant output. 


9.3.2.2 Uncertainty at Plant Input 


In the case where the modeling errors are reflected to the plant input, the actual plant is of the form 
Ga(s) = G(s)Z + wi(s)A(s)) (9.55) 


for some | A(jw) | 1. < 1. Following the procedure in Section 9.3.2, a sufficient condition for stability 
robustness is given by 


| wi Go) + K(jo)Gjo)) 1K (jo)GGo) |] 4, <1 (9.56) 


Using the definition of the H., norm and the fact that w;(s) is scalar, an equivalent condition for stability 
robustness is 


mal I + K(jo)G(jo))'Kjo)Ga)l < —~— Vo>0 (9.57) 
lwi(jo)| 

Comparison between the sufficient conditions Equations 9.54 and 9.57 reveals that, although the two 
conditions both address the same stability robustness problem, they are indeed different. When we reflect 
the modeling errors to the plant output, our stability robustness condition is based on the nominal 
complementary sensitivity TFM, C(s). This TFM is the closed loop TFM formed from the loop TFM with 
the loop broken at the plant output. On the other hand, when we reflect the modeling errors to the plant 
input, our stability robustness condition is based on the nominal input complementary sensitivity TFM, 
C,(s). In general, C;(s) is different from C(s). As mentioned in Section 9.2.3, this difference stems from 
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the fact that, although the sets G for the two cases both contain the actual plant, they are different because 
matrix multiplication does not commute. As a result, the two different conditions can give different 
results. Since the two conditions are both only sufficient, only one of them needs to be satisfied in order 
to conclude stability robustness. The fact that we only need to satisfy one of the conditions to achieve 
stability robustness means that one condition is less conservative than the other. This is indeed true since 
the sets G are different, and thus, invariably, one is larger than the other, resulting in a more conservative 
measure of modeling error. Finally, we note that if G(s) and K(s) are both SISO, then both Equations 9.54 
and 9.57 reduce to the sufficient condition derived for the SISO case. 


9.4 Impact of Stability Robustness on Closed-Loop Performance 


In the previous section, we have established, for both the SISO and MIMO cases, sufficient conditions 
for stability robustness under the multiplicative representation for unstructured uncertainty. To the 
control engineer, these conditions will have to be satisfied by design in order to ensure the stability 
of the actual closed-loop system. These conditions, then, become additional constraints on the design 
that will invariably impact the performance of the closed-loop system, such as command following and 
disturbance rejection. As shown in the chapter on MIMO frequency response analysis in this handbook, 
these performance specifications can be defined in the frequency domain in terms of the maximum 
singular value of the sensitivity TFM, S(s) = (I + G(s)K (s))~1. In particular, we require Omax[S(jw)] to 
be small in the frequency range of the command and/or disturbance signals. In this section, we seek 
an interpretation of our stability robustness conditions in the frequency domain in order to discuss its 
impact on command following and disturbance rejection. 

We start with the SISO case and the stability robustness condition given in Equation 9.48. Since 
everything is scalar and positive, this condition is equivalent to 


(9.58) 


| 8(jo)k(jo) 1 
1+g(jm)k(jw)| — |w(jo)| 


where the left side is the magnitude of the closed-loop or complementary sensitivity transfer function, 
c(jw), of the nominal design. This condition is illustrated in the frequency domain in Figure 9.17. 
Interpreting from the figure, the stability robustness condition states that the magnitude plot of the 
nominal closed-loop transfer function must lie strictly below the plot of the inverse of |w(jw)|. As is 
typically the case, the inverse of the magnitude of w( jw) will be large at low frequencies and small at high 
frequencies, since modeling errors increase with increasing frequency. Specifically, the modeling errors 
become significant near and above the frequency w, defined by 


|w(jom)| = 0 dB (9.59) 


Therefore, the stability robustness condition limits the bandwidth of the nominal closed-loop design. That 
is, the bandwidth of the nominal closed-loop design, wy», is constrained to be less than w,,, as shown in 
the figure. Indeed, as indicated in Figure 9.17, the presence of significant modeling errors at frequencies 
beyond ,, forces the rapid roll-off of the designed closed-loop transfer function at high frequencies. 
Physically, this roll-off prevents energy at these frequencies from exciting the unmodeled dynamics and, 
therefore, prevents the possible loss of stability. In terms of closed-loop performance, we recall from the 
chapter on MIMO frequency response analysis that a necessary condition for Omax[S(jw)] to be small 
at a certain frequency is that the singular values of the closed-loop or complementary sensitivity TFM, 
C(s) = G(s)K(s)(I + G(s)K(s))~!, must be close to unity (0 dB) at that frequency. For SISO systems, this 
simply translates to requiring that the magnitude of the closed-loop transfer function be close to unity 
(0 dB). From Figure 9.17, it is clear that the stability robustness condition limits the range of frequencies 
over which we can expect to achieve good command following and/or output disturbance rejection 
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FIGURE 9.17 Interpretation of stability robustness for SISO systems. 


to below w,,. In other words, we should not expect good performance at frequencies where there are 
significant modeling errors. 

For the MIMO case where the modeling errors are reflected to the plant output, the stability robustness 
condition given in Equation 9.54 can be illustrated in terms of singular value plots, as shown in Figure 9.18. 
This figure gives a similar interpretation concerning the impact of stability robustness on the nominal 
closed-loop performance as that for the SISO case. From the figure, the output bandwidth of the nominal 
closed-loop design, wy, is constrained by the stability robustness condition to be less than wy, where 
mo is defined as the frequency at and beyond which the modeling error reflected to the plant output 


a 
zZ 
3 1 
é |wo(ja)| 
sole) 

i} 
= 

OmaxlCJo)] 
0 


OminlCG )] 


FIGURE 9.18 Interpretation of stability robustness for MIMO systems with uncertainty reflected to plant output. 
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FIGURE 9.19 Interpretation of stability robustness for MIMO systems with uncertainty reflected to plant input. 


becomes significant. In other words, wo is defined as 
|Wo( j@mo)| = 0 dB (9.60) 


Therefore, as in the SISO case, the stability robustness condition limits the range of frequencies over which 
we can expect to achieve good command following and/or output disturbance rejection to below wno. 
On the other hand, for the MIMO case where the modeling errors are reflected to the plant input, we do 
not get a similar interpretation. In this case, the stability robustness condition given in Equation 9.57 can 
be illustrated in terms of singular value plots, as shown in Figure 9.19. The difference lies in the fact that 
this condition depends on the input complementary sensitivity TFM, Cj(s) = (I+ K (s)G(s))~!K(s)G(s), 
instead of C(s). Under this stability robustness condition, the input bandwidth of the nominal closed-loop 
design, wp;, is constrained to be less than w nj, where w,,; is defined as the frequency at and beyond which 
the modeling error reflected to the plant input becomes significant. Here, wi is defined as 


|wi( j@mi)| = 0 dB (9.61) 


which may be different than wo since w;(s) is, in general, not the same as w,(s). Since command following 
and output disturbance rejection depends on the sensitivity TFM, S(s) = (I + G(s)K (s))~!, instead of the 
input sensitivity TFM, S;(s) = (I + K(s)G(s))~!, we cannot directly ascertain the impact of the stability 
robustness condition in Equation 9.57 on these performance measures. However, we can obtain insight 
on the impact of this stability robustness condition on input disturbance rejection because the TFM from 
the disturbance at the plant input to the plant output is equal to G(s)(I + K(s)G(s))~!. Therefore, if the 
input sensitivity TFM is small then we have good input disturbance rejection. As before, a necessary 
condition for Omax[Sj(jw)] to be small at a certain frequency is that the singular values of C;( jw) are close 
to unity (0 dB). From Figure 9.19, this stability robustness condition clearly limits the range of frequencies 
at which we can expect to achieve good input disturbance rejection to below w nj. This is because we must 
roll-off Cj( jw) above w,,; to satisfy the stability robustness condition. 

To illustrate the concept of stability robustness to multiplicative unstructured uncertainty and its 
impact on closed-loop performance, we present the following example. 
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Example 9.2: Integrator with Time Delay 


Suppose that we wish to control the plant from Example 1 using a simple proportional controller. 
Our objective is to push the bandwidth of the nominal closed-loop system as high as possible. We 
use the simple integrator model 


g(s) = - (9.62) 


for our design. In this case, the nominal loop transfer function is simply g(s)k where k is the propor- 
tional gain of our controller, and the nominal closed loop transfer function is simply 


c(s) = —— (9.63) 


Since we require stability of the actual closed loop, we need to satisfy the stability robustness 
condition given in Equation 9 58 where, from Example 1, the weight w(s) of the uncertainty for this 
model is 

0.215 


w(s) = ——— 
0.055+1 


(9.64) 


As shown in Figure 9.20, the maximum value for k at which the stability robustness condition is 
satisfied is k = 5.9. With this value of k, the bandwidth achieved for the nominal system based on 
this model is 5.9 rad/s. 


Now, consider the case where the time delay can vary up to 0.4 s instead of 0.2 s. In this case, we use the 
same model as above, but our uncertainty representation must change in order to cover this additional 
uncertainty. Specifically, we must choose a new weight w(s). As in Example 1, we need to choose w(s) 
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FIGURE 9.20 Stability robustness condition for 0.2 s maximum delay uncertainty. 
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such that Ae 
|w(jo)| > max 2 sin (=)| Vo>0 (9.65) 


where t now ranges from 0 to 0.4 s. A simple w(s) that will satisfy Equation 9.65 is 


0.41s 
ary ee ie 
This is shown in Figure 9.21, where |w(jw)| and 2 | sin( -_ )| are plotted together on a magnitude Bode plot 
for t = 0.4, which is now the worst case value. We note that w(s) is proper and strictly stable as required. 

Since we require stability of the actual closed loop, we again need to satisfy the stability robustness 
condition given in Equation 9.58. As shown in Figure 9.22, the maximum value for k at which the stability 
robustness condition is satisfied is now k = 2.9. With this value of k, the bandwidth achieved for the 
nominal system based on this model is 2.9 rad/s. 

Comparing the two cases, it is clear that the uncertainty set for the second case is larger because the 
variation in the unknown time delay is larger. Physically, the larger time delay means larger unmodeled 
dynamics that we must cover in our uncertainty representation. This results in a larger magnitude for 
w(s) at low frequencies, as shown in Figure 9.23. In particular, the frequency at which the modeling errors 
become significant, w,, is lower for the second case than the first, and therefore, further limiting the 
achievable bandwidth of the nominal closed loop. As a result, we are able to push the bandwidth higher 
for the first case. This results in better achievable performance, such as command following and output 
disturbance rejection, as shown by the sensitivity bode plots in Figure 9.24. 

In summary, the requirement of stability robustness has a definite impact on the achievable nominal 
performance of the closed loop. This impact is typically a limitation on the nominal closed-loop bandwidth 
which, in turn, constrains the frequency range at which good command following and output disturbance 
rejection is achievable. As is evident in the above example, there is a trade-off between stability robustness 
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FIGURE 9.21 w(s) for 0.4 s maximum delay uncertainty. 
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FIGURE 9.22 Stability robustness condition for 0.4 s maximum delay uncertainty. 
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FIGURE 9.23 Comparing |w(jw)|. 
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FIGURE 9.24 Comparing sensitivity magnitude plots. 


and performance. That is, a controller that is designed to be robust to a larger uncertainty set will 
result in poorer performance due to the fact that its bandwidth will be further limited. In addition, 
there is a connection between this trade-off and the conservatism of our stability robustness conditions. 
In particular, our representation of the uncertainty is one of overbounding the set of TFMs from our 
modeling process with a larger set. If we choose w(s) where the overbounding is not tight, then this will 
further limit the amount of achievable performance that we can obtain for the closed loop. Therefore, we 
see that both the concept of unstructured uncertainty and the way we represent this uncertainty in the 
stability robustness problem have an impact on achievable nominal performance. As a result, to improve 
performance we may have to reduce the level of uncertainty by having a better model and/or by obtaining 
a better representation for the uncertainty. Finally, we wish to point out that only nominal performance or 
the closed-loop performance of the model without the uncertainty is discussed. The actual performance 
of the loop will be different. However, for low frequencies where we care about command following 
and output disturbance rejection, the modeling errors will be typically small. Therefore, the difference 
between actual performance and nominal performance at these frequencies will be small. 


9.5 Other Representations for Model Uncertainty 


In this section, we present some alternative set membership representations for unstructured uncertainty. 
For each representation, we will start with the definition of the set G. In defining G, we assume throughout 
that 


1. The weight is a fixed, proper, and strictly stable scalar transfer function 
2. A(s) isa strictly stable TFM 
3. No unstable or imaginary axis poles of G(s) are cancelled in forming any element G(s) of the set G 


With this definition, the appropriate stability robustness condition is obtained using the procedure 
outlined in Section 9.3.2. Like those derived for the multiplicative uncertainty representation, these 
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conditions are all conservative since they are derived using the small gain theorem. The focus here will 
be primarily on the presentation of results rather than detailed treatments of their derivation. MIMO 
systems are treated as SISO systems that are simply special cases. Comments on the usefulness of these 
representations and interpretations of their results are given as appropriate. 


9.5.1 Additive Uncertainty 


An alternative representation for unstructured uncertainty is additive uncertainty. In this case, the set G 
is defined as 
G = {G(s) | G(s) = G(s) + wa(s)A(s), | AGio) lla. = UB (9.67) 


Following the procedure in Section 9.3.2, the stability robustness condition for additive uncertainty is 


Omaxl (I + K(jw)G(jo)) 1K (jw)] < aio Yo>o (9.68) 


From Equation 9.67, the additive error is defined as 
Eq(s) = wa(s)A(s) = G(s) — G(s) (9.69) 


which is simply the difference between the perturbed plant and the model. As a result, the additive 
representation is a more natural representation for differences in the internal dynamics between the actual 
plant and the model. In particular, it is no longer necessary to reflect these modeling errors to the plant 
input or output. However, we note that the resulting stability robustness condition is explicitly dependent 
on the controller TFM, K(s), and the loop TFM, K(s)G(s), instead of simply on K(s)G(s). This is because, 
unlike multiplicative uncertainty, the uncertainty representation here does not apply equally to the loop 
TFM as to the model, G(s). The result is that there will be added complexity in designing a K(s) that will 
satisfy the condition because shaping the TFM, (I + K(jw)G(jw))~!K (jw), to satisfy Equation 9.68 will 
require shaping both K(jw) and K(jw)G(jw). Due to this complication, the multiplicative uncertainty 
or another form of cascaded uncertainty representation, where the representation applies equally to G(s) 
as to K(s)G(s), is often used instead in practice. 


9.5.2 Division Uncertainty 


Another representation for unstructured uncertainty is the division uncertainty. Like the multiplicative 
uncertainty representation, the division uncertainty represents the modeling error in a cascade form, and 
therefore, the modeling error can be reflected either to the plant output or the plant input. In the case 
where the modeling errors are reflected to the plant output, the set G is defined as 


G = {G(s) | G(s) = I + wao(s)A(s)) G(s); | AG@) lla, < 1 (9.70) 


Following the procedure in Section 9.3.2, the stability robustness condition for division uncertainty at the 
plant output is given by 


Omaxl(I + G(jw)K(jo)) 1] < ee Vw>0 (9.71) 
|Wdo( jo)| 


Similarly, for the case where the modeling errors are reflected to the plant input, the set G is defined as 
G = {G(s) | G(s) = Gs)(I+ wai(s)A(S))' JAG) Il. = (9.72) 


and the resulting stability robustness condition is given by 


Omaxl(I + K(jw)G(jo))!] < a Yw>0 (9.73) 
|wai(jo)| 
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FIGURE 9.25 Interpretation of stability robustness for division uncertainty at plant output. 


As shown in Figure 9.25, the stability robustness condition for division uncertainty at the plant output 
requires the maximum singular value plot for the sensitivity TFM of the nominal system, Omax[S(jw)], to lie 
strictly below the plot of the inverse of |wgo( jw)|. Similarly, for division uncertainty at the plant input, the 
stability robustness condition requires that the maximum singular value plot for the input sensitivity TFM 
of the nominal system, Omax[S;(j@)], must lie strictly below the plot of the inverse of | wgj(jw)|. As shown in 
the figure, the inverse of the magnitude of wg9(s) is typically large at low frequencies and approaches unity 
(0 dB) at high frequencies where the modeling errors become significant. Since the stability robustness 
condition for division uncertainty at the plant output depends on the nominal sensitivity TFM, its impact 
on command following and output disturbance rejection is clear from Figure 9.25. In particular, if the 
modeling error becomes significant at a particular frequency, wm. such that the plot of the inverse 
of |Wdo(jw)| is close to 0 dB at that frequency and beyond, the control must be designed such that 
Omax[S(jw)] is below this barrier. That is, Omax[S(jw)] is not allowed to be much greater than 0 dB 
for w > Wmo. However, we know from the Bode integral theorem [4] that if we suppress the sensitivity 
at high frequencies then we cannot also suppress it at low frequencies. Therefore, we again see how 
modeling errors can place a limitation on the range of frequencies over which we can expect to achieve 
good command following and output disturbance rejection. The same can be said for input disturbance 
rejection with respect to the stability robustness condition for division uncertainty at the plant input. 


9.5.3 Representation for Parametric Uncertainty 
Finally, we can also define the set G as 
G = {G(s) | G(s) = (I + wp(s)G(s)A(s))* G(s), |ACjo) Il, < (9.74) 


In this case, the stability robustness condition is given by 


Omaxl (I + G(jo)K(jo))'G(ja)] < (9.75) 


pee = 
|wp(jo)| 


As shown in the following example, we can use this representation to handle parametric uncertainty in 
the A matrix of the nominal model G(s). 
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FIGURE 9.26 Representation for parametric uncertainty in the A matrix. 


Example 9.3: Parametric Uncertainty in the A Matrix 


Consider the dynamics of the actual plant in state-space form 


xX =Ax+ Bu 
(9.76) 
y= 
where the parameters of the A matrix are uncertain but are known to be constant and contained in 
a certain interval. In particular, consider the case where the A matrix is given as 


A=A+38A (9.77) 


where the elements of A contain the midpoint values for each corresponding element in A and each 
element of 8A is known to exist in the interval 


-l1< dAjj <1 (9.78) 


We note that since 
|SAl| <n (9.79) 


for all possible 8A satisfying Equation 9.78 where n is the dimension of A, it is clear that the set of 
possible 8A is contained in the set {wp(s) A(s)| IAG@)lla1., < 1, Wp(s) = n}. Therefore, we can express 
the A matrix as 

A=A+wo(s)A(s) (9.80) 
for Wp(s) =n and for some [ACGo@)llaz., <1. In block diagram form, the state-space equations in 
Equation 9.76 can then be represented as in Figure 9.26, where the model G(s) is taken to be 
(s! — A)—!. We note that this is equivalent to the uncertainty representation given in Equation 9.74. 


We note that the above representation for parametric uncertainty is, in general, very conservative. This 
is because the representation allows for complex perturbations of the matrix A since A(s) is complex. 
Since the parameters are real, the perturbed set of plants is much larger than the set that will actually be 
realized by the plant, which results in the conservatism. 


Notes 


The majority of the material in this article is adopted from Doyle and Stein [5], Dahleh and Diaz- 
Bobillo [1], Maciejowski [6], and Doyle et al. [7]. Other excellent references include Green and Lime- 
beer [8] and Morari and Zafiriou [9]. 
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10.1 Introduction and Motivation 


It is well known that feedback may be used in a control design to obtain desirable properties that are 
not achievable with open-loop control. Among these are the ability to stabilize an unstable system and 
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to reduce the effects of plant disturbances and modeling errors upon the system output. On the other 
hand, use of feedback control can also have undesirable consequences: feedback may destabilize a system, 
introduce measurement noise, amplify the effects of disturbances and modeling errors, and generate large 
control signals. A satisfactory control design will, if possible, achieve the potential benefits of feedback 
without incurring excessive costs. 

Unfortunately, feedback systems possess limitations that manifest themselves as design trade-offs 
between the benefits and costs of feedback. For example, there exists a well-known trade-off between 
the response of a feedback system to plant disturbances and to measurement noise. This trade-off is an 
inherent consequence of feedback system topology; indeed, a plant disturbance cannot be attenuated 
without a measurement of its effect upon the system output. Other design trade-offs are a consequence 
of system properties such as unstable poles, nonminimum phase zeros, time delays, and bandwidth 
limitations. 

The study of feedback design limitations and trade-offs dates back at least to the seminal work of 
Bode [7]. In his classic work, Bode stated the famous gain-phase relation and analyzed its implications for 
the classical loop-shaping problem. He also derived the Bode sensitivity integral. The importance of the 
sensitivity integral to feedback control design was emphasized by Horowitz [36]. Connections between 
the classical loop-shaping problem, including the gain-phase relation, and modern control techniques 
were developed by Doyle and Stein [22]. Further results pertaining to design limitations were derived 
by Freudenberg and Looze [28,31], who studied inherent design limitations present when the system to 
be controlled has unstable poles, nonminimum phase zeros, time delays, and/or bandwidth constraints. 
A result dual to the Bode sensitivity integral, and applicable to the complementary sensitivity function, 
was obtained by Middleton and Goodwin [43,45]. In all these works, emphasis is placed upon frequency 
response properties and, in particular, the insight into feedback design that may be obtained from a Bode 
plot. 

In this chapter, we describe several trade-offs that are present in feedback system design due to the 
structure of a feedback loop, bandwidth limitations, and plant properties such as unstable poles, nonmini- 
mum phase zeros, and time delays. We focus primarily on linear time-invariant single input single output 
systems (SISOs) and later give a brief description and references to a number of extensions of these 
results. In Section 10.2, we show how the closed-loop transfer functions of a feedback system may be 
used to describe signal response, differential sensitivity, and stability robustness properties of the system. 
In Section 10.3, we describe design trade-offs imposed by the topology of a feedback loop and the relation 
between feedback properties and open-loop gain and phase. Next, in Section 10.4, we describe design 
limitations imposed by the Bode gain-phase relation, Bode sensitivity integral, and the dual complemen- 
tary sensitivity integral. These trade-offs are present due to the fact that the closed-loop system is stable, 
linear, and time invariant. When the plant has nonminimum phase zeros, unstable poles, and/or time 
delays, additional design trade-offs are present. As discussed in Section 10.5, these may be described using 
the Poisson integral. An alternate view of some of the performance limitations using time responses is 
described in Section 10.6. In Section 10.7, we give a brief summary and references to various extensions 
of the theory of feedback performance limitations, including discrete-time systems, systems with feed- 
back over a communication channel, limits on optimal cheap control and H2 performance, multivariable 
systems, and nonlinear and time-varying systems. We conclude the section with a brief discussion of 
cases in which performance limitations may be alleviated in tracking problems. A general summary and 
suggestions for further reading are presented in Section 10.8. 


10.2 Quantification of Performance for Feedback Systems 
10.2.1 Overview 


Throughout this chapter, we consider the single-degree-of-freedom unity feedback system depicted in 
Figure 10.1. In this figure, P(s) and C(s) denote the transfer functions of the plant and the compensator, 
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FIGURE 10.1 Linear time-invariant feedback system. 


respectively. The system output is denoted y(t), the control signal by u(t), and the command input by 
r(t). An important goal in many feedback control problems is that of forcing y(t) to track r(t) despite the 
presence of input and output disturbances d;(t) and do(t). To do so, a measurement of the system output 
is compared to the command input to form an error signal e(t), and used as an input to the controller. 
The effects of measurement noise are denoted n(t). 

Other feedback architectures are possible, including a two-degree-of-freedom architecture, in which 
the command input and measured output are processed separately. For example, the command input 
may be shaped by a precompensator before the error signal is formed. Similarly, any disturbances that 
are measurable may also be processed separately. Throughout this chapter, we discuss the feedback 
architecture in Figure 10.1, and assume that any precompensation is absorbed into the definition of the 
command input r(t). 

The ability of the plant output to follow the command input is referred to as tracking. In order that 
tracking be feasible, the closed-loop system must, at a minimum, be stable. As Figure 10.1 illustrates, 
exogenous signals can act at various points in the feedback loop to affect the response of the system. 
Signals that enter the loop between the control action and the output to be controlled are referred to as 
disturbance signals. The ability of the control system to eliminate the effects of the disturbances is referred 
to as disturbance rejection. Signals that enter the loop between the system output and the comparison of 
the measurement with the command input are referred to as measurement noise signals. Measurement 
noise only affects the output through the action of the compensator. 

In addition to the uncertainty caused by the disturbance signals and measurement noise, the behavior 
of the plant as described by the model will differ from that of the true plant. A feedback system that is 
stable for the plant model is called nominally stable. A feedback loop that maintains stability when subject 
to variations in the plant model is robustly stable. Nominal performance is achieved when tracking 
and disturbance rejection objectives are met for the plant model. Robust performance is achieved if the 
feedback loop maintains performance (tracking or disturbance rejection) when subject to variations in 
the plant model. 

To proceed with our analysis, we define closed-loop transfer functions from the exogenous input 
signals in Figure 10.1 to the system output, measured error, and the control signal, and shall quantify 
performance in terms of the peak in a Bode gain plot of each transfer function. We shall also see that these 
closed-loop transfer functions describe the sensitivity and robustness properties of a feedback system. 


10.2.2 Nominal Signal Response 


Consider the linear time-invariant feedback system shown in Figure 10.1. Three transfer functions are 
essential in studying properties of this system. Define the open-loop transfer function 


L(s) = P(s)C(s), (10.1) 


the sensitivity function 


S(s) (10.2) 


= 1 
~ 14 L(s)’ 
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and the complementary sensitivity function 


L(s) 


= Tee 


(10.3) 


The response of the system of Figure 10.1 to the exogenous inputs (i.e., the command input, the 
input disturbance, the output disturbance, and the sensor noise) is governed by the closed-loop transfer 
functions (Equations 10.2 and 10.3). The output of the system is given by 


¥(s) = Ss) (Do(s) + P(s)Di(s)) + T() (ROS) — N(S)) (10.4) 
The feedback loop error between the measured output and the command input is given by 
E(s) = S(s) (R(s) — Do(s) — P(s)Di(s) — N(s)) . (10.5) 
The performance error between the output and the command input is given by 
Ep(s) = R(s) — Y(s) = S(s) (R(s) — Do(s) — P(s)Di(s)) — T(s)N(s). (10.6) 
The control input generated by the compensator is given by 
U(s) = —T(s)Di(s) + S(s)C(s) (R(s) — N(s) — Do(s)) - (10.7) 


Because the principal objective for control system performance is to have the plant output track 
the command input without using too much control energy, it is desirable that the performance error 
(Equation 10.6) and the plant input (Equation 10.7) be small. It is apparent from Equations 10.6 and 10.7 
that each of these signals can be made small by making the sensitivity and complementary sensitivity 
transfer functions small. In particular, the performance error (Equation 10.6) can be made small at 
any frequency s = jw by making the sensitivity function small relative to the command response and 
disturbances, and by making the complementary sensitivity small relative to the measurement noise. The 
plant input can also be kept within a desired range by bounding the magnitude of the complementary 
sensitivity function relative to the external signals. The presence of the plant transfer function in the 
denominator of the second term in Equation 10.7 implies that the magnitude of the control signal will 
increase as the plant magnitude decreases, unless there is a corresponding decrease in the magnitudes of 
either the complementary sensitivity function or the external command, measurement noise, and output 
disturbance signals. The presence of high-frequency measurement noise, or disturbances together with a 
bound on the desired control magnitude thus imposes a bandwidth constraint on the closed-loop system. 


10.2.3 Differential Sensitivity 


Because the true plant behavior will be somewhat different from the behavior described by the plant model 
P(s), it is important that performance be maintained for variations in the plant model that correspond to 
possible differences with reality. Although the precise characterization of robust performance is beyond 
the scope of this chapter (see [23]), incremental changes in the system response due to model variations 
can be characterized by the sensitivity function. Assume that the true plant model is given by 


P’(s) = P(s) + AP(s), (10.8) 


where AP(s) represents the difference between the true and nominal values of the plant transfer function. 
Assuming that the disturbance and measurement noise signals are zero, the nominal closed-loop response 
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(i.e., the response to the command reference for the plant model P(s)) is 
Y(s) = T(s)R(s). (10.9) 


The response for the feedback system using the true plant is 


Y'(s) = T’(s)R(s), (10.10) 

where 
Y'(s) = Y(s) + AY(s), (10.11) 
T'(s) = T(s) + AT(s). (10.12) 


Thus the variation in the command response is proportional to the variation in the true complementary 
sensitivity function. 
Let S’(s) denote the sensitivity function for the feedback system using the true plant transfer function. 
Then, it can be shown that [20] 
AT(s) — 9(s) ANG) 
T(s) P(s) 


Equation 10.13 shows that the true sensitivity function determines the relative deviation in the command 
response due to a relative deviation in the plant transfer function. If the true sensitivity function is small, 
the variation in command response will be small relative to the variation in the plant model. 

Although Equation 10.13 provides insight into how the command response varies, it is not useful as a 
practical analysis and design tool because the relationship is expressed in terms of the unknown true plant 
transfer function. However, as the plant variation AP(s) becomes small, the true sensitivity function S’(s) 
approaches the nominal sensitivity function S(s). Thus, for AP(s) = dP(s), Equation 10.13 becomes 


(10.13) 


dT(s)__— dP(s) 
To) 9 Be" 


(10.14) 


Equation 10.14 shows that the sensitivity function S(s) governs the sensitivity of the system output 
to small variations in the plant transfer function. Hence, the variation in the command response will be 
small if the sensitivity function is small relative to the plant variation. 


10.2.4 Robust Stability 


The feedback system in Figure 10.1 will be internally stable if each transfer function S(s), T(s), S(s)P(s), 
and C(s)S(s) is proper and have no poles in the closed right half-plane (CRHP) [34]. If there are no 
CRHP pole-zero cancellations between the plant and the controller, then the feedback system will be 
stable if the sensitivity function is proper and has no CRHP poles. A number of techniques, such as the 
Routh-Hurwitz and Nyquist criteria [21,27], are available for evaluating whether the system is stable for 
the nominal plant model. A more challenging problem is to determine whether the feedback system will 
be robustly stable for the true but unknown plant. 

The sensitivity and complementary sensitivity functions each characterize stability robustness of the 
system against particular classes of plant variations (e.g., see [70], Table 9.1). One of the most significant 
types of plant modeling error is the high-frequency uncertainty that is present in any model of a physical 
system. All finite-order transfer function models of physical systems deteriorate at high-frequencies 
due to neglected dynamics (such as actuator and measurement lags, and flexible modes), the effects of 
approximating distributed systems with lumped models, and time delays. The result of these neglected 
physical phenomena is that the uncertainty in the gain of the true plant increases (eventually differing 
from the assumed plant model by more than 100%) and the phase of the true plant becomes completely 
uncorrelated with that of the model (i.e., there is +180° uncertainty in the phase). 
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This type of uncertainty can be represented as a relative deviation from the nominal plant. The true 
plant transfer function is represented by a multiplicative error model 


P’(s) = P(s)(1+ A(s)). (10.15) 
Given a true plant P’(s), the multiplicative error A(s) is the relative plant error 


MOS P — 


(10.16) 
It will be assumed that the multiplicative error is stable. From Equation 10.16, it is apparent that this 
assumption will be satisfied if the number and location of any unstable poles of the true plant are the 
same as those of the plant model. 

The characteristics of increasing gain and phase uncertainty can be represented by assuming that the 
multiplicative error is unknown except for an increasing upper bound on its magnitude. That is, it will be 
assumed that A(s) is any stable transfer function that satisfies 


|A(jw)| < Ma(o), (10.17) 


where (typically) Ma(@) > 00 as w > 00. 

If the only information available about the uncertainty is Equation 10.17, then A(s) is referred to as 
unstructured uncertainty. In that case, a necessary and sufficient condition for the feedback system to be 
stable for all plants described by Equations 10.15 through 10.17 is that the system be stable when A(s) = 0 
and that the complementary sensitivity function satisfy the bound [22] 


|T(jw)| < Vo. (10.18) 


1 
Ma(o)’ 
Equation 10.18 demonstrates that the presence of high-frequency uncertainty forces the complementary 
sensitivity function to become small to insure that the system is robustly stable. This in turn implies that 
an upper limit on bandwidth is imposed on the closed-loop system. 


10.2.5 Summary of Feedback System Performance Specification 


In the previous section we saw that the sensitivity and complementary sensitivity functions each charac- 
terize important properties of a feedback system. This fact motivates stating design specifications directly 
as frequency-dependent bounds upon the magnitudes of these functions. Hence, we typically require that 


|SGjo)| < Ms(w), Vo, (10.19) 


and 
ITGio)|<Mr(@), Vo. (10.20) 


The bounds Ms(w) and Mr(o) will generally depend on the size of the disturbance and noise signals, the 
level of plant uncertainty, and the extent to which the effect of these phenomena upon the system output 
must be diminished (see Equations 10.6, 10.7, 10.14, and 10.18). 

It is interesting to note that the two design specifications represent two different aspects of control 
system design. The bound (Equation 10.19) on the sensitivity function typically represents the potential 
benefits of feedback, such as disturbance rejection, tracking and robust performance. The bound (Equa- 
tion 10.20) on the complementary sensitivity function represents desired limits on the cost of feedback: 
amplification of measurement noise, increased control signal requirements, and possible introduction of 
instability into the feedback loop. 
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10.3 Design Trade-Offs at a Single Frequency 


10.3.1 Introduction 


Often the requirements imposed by various design objectives are mutually incompatible. It is therefore 
important to understand when a given design specification is achievable and when it must be relaxed 
by making trade-offs between conflicting design goals. The objective of this section is to explore the 
limitations that are imposed by the structure of the feedback loop in Figure 10.1. 

The structural trade-offs presented in this section express the limitation that there is only one degree 
of design freedom in the feedback loop. This degree-of-freedom can be exercised by specifying any one 
of the associated transfer functions. Because the sensitivity and complementary sensitivity functions are 
both uniquely determined by the loop transfer function, these transfer functions cannot be specified 
independently. Hence, the properties of the feedback loop are completely determined once any one of 
L(s), S(s), or T(s) are specified, and these properties can be analyzed in terms of the chosen transfer 
function. In particular, the performance bounds (Equations 10.19 and 10.20) are not independent, and 
may lead to conflicts. 

This section presents explicit relationships between the sensitivity function, the complementary sensi- 
tivity function, and the loop transfer function of the feedback system of Figure 10.1. These relationships 
can be used to explore whether design specifications in the form of (Equation 10.19 and 10.20) are 
consistent, and if not, how the specifications might be modified to be achievable. 


10.3.2 Relationship between Closed-Loop Transfer Functions 
and Design Specifications 


One important design trade-off may be quantified by noting that the sensitivity and complementary 
sensitivity functions satisfy the identity 


S(jo) + T(jo) = 1. (10.21) 


Equation 10.21 is a structural identity in the sense that it is a consequence of the topology of the feedback 
loop. 

It follows from Equation 10.21 that |S(j@)| and |T(jo)| both cannot be very small at the same fre- 
quency. Hence, at each frequency, there exists a trade-off between those feedback properties such as 
sensitivity reduction and disturbance response that are quantified by |S(jw)| and those properties such as 
measurement noise response and robustness to high-frequency uncertainty that are quantified by |T(jw)|. 

In applications it often happens that levels of uncertainty and sensor noise become significant at 
high frequencies, while disturbance rejection and sensitivity reduction are generally desired over a lower 
frequency range. Hence, the trade-off imposed by Equation 10.21 is generally performed by requiring 
Ms(@) to be small at low frequencies, My(w) to be small at high frequencies, and neither Ms(w) nor 
Mr() to be excessively large at any frequency. 

This situation is illustrated in Figure 10.2. The bound Ms(w) is small at low frequencies, representing 
the frequencies over which disturbance rejection and tracking are important. It increases with increasing 
frequency until it becomes greater than one at high frequencies. Conversely, the bound Mr(w) begins 
somewhat greater than one at low frequencies, and rolls off (decreases in magnitude) at higher frequencies 
where the plant model error and measurement noise become significant. The sensitivity and complemen- 
tary sensitivity functions shown in Figure 10.2 satisfy the design objectives (Equations 10.19 and 10.20). 

Equation 10.21 can be used to derive conditions that the design objective functions Ms(w) and Mr(w) 
must satisfy. Taking the absolute value of the right-hand side of Equation 10.21, using the triangle inequal- 
ity and applying the bounds on the sensitivity and complementary sensitivity functions (Equations 10.19 
and 10.20), the following inequality is obtained: 


Ms(w) + Mr(w) > 1. (10.22) 
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FIGURE 10.2 Typical sensitivity function, complementary sensitivity function, and specification bounds. 


Equation 10.22 reinforces the observations made previously: the design objectives cannot be specified to 
require both the sensitivity function and the complementary sensitivity function to be small at the same 
frequency. 

The structural identity can also be used to explore limits on more detailed specifications of control 
system objectives. For example, suppose that at a given frequency w the dominant objective that specifies 
the weighting on the complementary sensitivity function is that the control signal should remain bounded 
when subject to an output disturbance which is unknown but bounded: 


|UGjo)| < Mi(), V|Do(jw)| < Ma,(). (10.23) 


Then, combining Equations 10.7 and 10.23, the design specification on the complementary sensitivity 
Equation 10.20 becomes 

Miu(o) = 
Ma, () 
Substituting this value for the design specification bound on the complementary sensitivity into the 
inequality (Equation 10.22) yields 


|T(jo)| < |PCj@)| Mr(o). (10.24) 


M,,() 
Mga,() 


Ms(w) + |PUjo)| 


IV 


i: (10.25) 


Inequality (Equation 10.25) requires the design specification on the sensitivity function to increase as the 
plant magnitude decreases relative to the available control authority for a given disturbance. Thus, the 
desire to reject disturbances (reflected by smaller Ms(w)) is limited. The trade-off between disturbance 
rejection and control amplitude is quantified by Equation 10.25. 


10.3.3. Relation between Open and Closed-Loop Specifications 


Because specifying one of the transfer functions S(s), T(s), or L(s) completely determines the others, any 
one of them can be used as a basis for analysis and design. Classical “loop-shaping” design methods 
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proceeded by directly manipulating the loop transfer function L(s) (using, e.g., lead and lag filters) to alter 
the feedback properties of the system. As is well known, these methods are quite effective in coping with 
the type of design problems for which they were developed. One reason for the success of these methods 
is that, for a scalar system, open-loop gain and phase can be readily related to feedback properties. Indeed, 
the following relations are well known (e.g., [22,36]) and can readily be deduced from Equations 10.2 and 
10.3: 


|L(jm)|>>1 = |SGjw)| «1 and T(jw) ~ 1 (10.26) 


and 
IL(jo)|K1 = S(jw)*1 and |T(jo)| «1. (10.27) 


At frequencies for which open-loop gain is approximately unity, feedback properties depend critically 
upon the value of open-loop phase: 


|L(jm)| © 1 and ZL(jw) ¥ +180° = |S(jw)|>>1 and |T(ja)| > 1. (10.28) 


These approximations (Equations 10.26 through 10.28) yield the following rules of thumb useful in 
design. First, large loop gain yields small sensitivity and good disturbance rejection properties, although 
noise appears directly in the system output. Second, small loop gain is required for small noise response 
and for robustness against large multiplicative uncertainty. Finally, at frequencies near gain crossover 
(|L(jw)| © 1), the phase of the system must remain bounded sufficiently far away from +180° to provide 
an adequate stability margin and to prevent amplifying disturbances and noise. 

It is also possible to relate open-loop gain to the magnitude of the plant input. From Equations 10.26 
and 10.2, it follows that 


|L(jo)| > 1 = |S(jw)C(ja)| © |P~!(ja)|. (10.29) 


Hence, requiring loop gain to be large at frequencies for which the plant gain is small may lead to 
unacceptably large response of the plant input to noise and disturbances (see also Equation 10.25). 

Recall the discussion of the requirements imposed on the magnitudes of the sensitivity and comple- 
mentary sensitivity by the performance objectives (Equations 10.19 and 10.20) (see Figure 10.2). From that 
discussion and the approximations (Equations 10.26 through 10.28), it follows (cf., [22]) that correspond- 
ing specifications upon open-loop gain and phase might appear as in Figure 10.3. These specifications 
reflect the fact that loop gains must be large at low frequencies for disturbance rejection and sensitivity 
reduction and must be small at high frequencies to provide stability robustness. At intermediate frequen- 
cies the phase of the system must remain bounded away from +180° to prevent excessively large values 
of |S(jw)| and |T(jw)|. To obtain the benefits of feedback over as large a frequency range as possible, 
it is also desirable that w, be close to wy. Of course, gain and phase must also satisfy the encirclement 
condition dictated by the Nyquist stability criterion. 


4 |LG@)| 4 ZL(jo) 
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functions (0) phase functions (0) 
> > 
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FIGURE 10.3 Gain and phase specifications. 
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10.4 Constraints Imposed by Stability 


10.4.1 Introduction 


Implicit in our construction of design specifications such as those illustrated by Figures 10.2 and 10.3 is 
the assumption that the transfer functions can be prescribed independently in different frequency regions 
or that open-loop gain and phase can be independently manipulated. However, the requirement that the 
closed-loop system be stable imposes additional constraints on the transfer functions that relate the values 
of the functions in different frequency regions. These constraints will be referred to as analytic constraints 
because they result from the requirement that the transfer functions have certain analytic properties. 

As has been noted in Section 10.3, any of the transfer functions S(s), T(s), or L(s) can be used to char- 
acterize the performance and properties of the feedback system. This section will present the constraints 
imposed by stability for each of these transfer functions. 


10.4.2 Bode Gain-Phase Relation and Interpretations 


We have shown in the previous section how classical control approaches view design specifications in 
terms of limits on the gain and the phase of the open loop transfer function. However, the gain and phase 
are not mutually independent: the value of one is generally determined once the other is specified. There 
are many ways to state this relationship precisely; the one most useful for our purposes was derived by 
Bode [7]. This relation has been used by many authors (cf. [22,36]) to analyze the implications that the 
gain-phase relation has upon feedback design. 


Theorem 10.1: Bode Gain-Phase Relation 


Assume that L(s) is a rational function with real coefficients and with no poles or zeros in the CRHP. Then, 
at each frequency s = joo, the phase of L(s) must satisfy the integral relation* 


. 1 (® dlog|L| |v| 
ZL — ZL(0) = — 1 th —dv, 10.30 
(joo) (0) ~f Fy ogcoth = dv (10.30) 
where 
v= log (=) (10.31) 
9 


Theorem 10.1 states conditions (i.e., L(s) rational, stable, and minimum phase) under which knowledge 
of open-loop gain along the jw-axis suffices to determine open-loop phase to within a factor of +n. 
Constraints analogous to Equation 10.30 hold for unstable or nonminimum phase systems. Hence, gain 
and phase cannot be manipulated independently in design. 

Equation 10.30 shows that the phase of a transfer function is related to the slope (on a log-log scale) 
of the magnitude of the transfer function. The presence of the weighting function log coth(|v|/2) = 
log |(@ + w9)/(@ — wo)| shows that the dependence of ZL(jw 9) upon the rate of gain decrease at frequency 
w diminishes rapidly as the distance between w and wo increases (see Figure 10.4). Hence this integral 
supports a rule of thumb stating that a 20N dB/decade rate of gain decrease in the vicinity of frequency 
wo implies that ZL(jwo) * —90N°. In many practical situations, the open loop transfer function is 
sufficiently well behaved that the value of phase at a frequency is largely determined by that of the gain 
over a decade-wide interval centered at the frequency of interest [7]. 

The Bode gain-phase relation may be used to assess whether a design specification of the type shown in 
Figure 10.3 is achievable. Since a 20N dB/decade rate of gain decrease in the vicinity of crossover implies 


* Throughout this article, the notation log(-) will be used to denote the natural logarithm. 
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FIGURE 10.4 Weighting function in gain-phase integral. 


that phase at crossover is roughly —90N°, it follows that the rate of gain decrease cannot be much greater 
than 20 dB/decade if the Nyquist stability criterion is to be satisfied and if an acceptable phase margin is 
to be maintained. One implication of this fact is that the frequency w, in Figure 10.3 cannot be too close 
to the frequency wy. Hence, the frequency range over which loop gain can be large to obtain sensitivity 
reduction is limited by the need to ensure stability robustness against uncertainty at higher frequencies, 
and to maintain reasonable feedback properties near crossover. As discussed in [22,36], relaxing the 
assumption that L(s) has no right-half-plane poles or zeros does not lessen the severity of this trade-off. 
Indeed, the trade-off only becomes more difficult to accomplish. 


10.4.3 The Bode Sensitivity Integral 


The purpose of this section is to present and discuss the constraint imposed by stability on the sensitivity 
function. This constraint was first developed in the context of feedback systems in [7]. This integral 
quantifies a trade-off between sensitivity reduction and sensitivity increase that must be performed 
whenever the open-loop transfer function has at least two more poles than zeros. 

The magnitude of the sensitivity function of a scalar feedback system can be obtained easily using a 
Nyquist plot of L(jw). Indeed, since S(jw) = 1/(1+ L(jw)), the magnitude of the sensitivity function is 
equal to the reciprocal of the distance from the Nyquist plot to the critical point. In particular, sensitivity 
is less than one at frequencies for which L(jw) is outside the unit circle centered at the critical point. 
Sensitivity is greater than one at frequencies for which L(jq) is inside this unit circle. 

To motivate existence of the integral constraint, consider the open-loop transfer function L(s) = 
2/(s+ 1). As shown in Figure 10.5, there exists a frequency range over which the Nyquist plot of L(jw) 
penetrates the unit circle centered at —1 + j0 and sensitivity is thus greater than one. In practice, the open- 
loop transfer function will generally have at least two more poles than zeros [36]. If L(s) is stable, then, 
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Nyquist plot of L(s) = 2/(s+1)” 
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FIGURE 10.5 Effect of a two-pole rolloff upon the Nyquist plot of L(s) = 2/(s+ 1)?. 


using the gain-phase relation (Equation 10.30), it is straightforward to show that L(jw) will asymptotically 
have a phase lag of at least —180°. Hence there will always exist a frequency range over which sensitivity 
is greater than one. This behavior may be quantified using a classical theorem due to Bode [7], which was 
extended in [31] to allow unstable poles in the open-loop transfer function. An excellent discussion of the 
implications that the Bode sensitivity integral has for open-loop unstable systems can be found in [62]. 


Theorem 10.2: Bode Sensitivity Integral 


Suppose that the open-loop transfer function L(s) is rational and has right-half-plane poles 
| etl... viNp}s with multiple poles included according to their multiplicity. If L(s) has at least two 
more poles than zeros, and if the associated feedback system is stable, then the sensitivity function must 


satisfy 
ioe) Np 
/ log |S(jw)|dw = x)” Re(pi). (10.32) 
0 


i=1 


It is possible to extend Theorem 10.2 to systems for which L(s) has relative degree one and a possible 
time delay. For details, see [56, Theorem 3.1.4]. 

Theorem 10.2 shows that a trade-off exists between sensitivity properties in different frequency ranges. 
Indeed, for stable open-loop systems, the area of sensitivity reduction must equal the area of sensitivity 
increase on a plot of the logarithm of the sensitivity versus linear frequency (see Figure 10.6). In this 
respect, the benefits and costs of feedback are balanced exactly. 

The extension of Bode’s theorem to open-loop unstable systems shows that the area of sensitivity 
increase exceeds that of sensitivity reduction by an amount proportional to the distance from the unstable 
poles to the left half-plane. A little reflection reveals that this additional sensitivity increase is plausible for 
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the following reason. When the system is open-loop unstable, then it is obviously necessary to use feedback 
to achieve closed-loop stability, as well as to obtain sensitivity reduction. One might expect that this 
additional benefit of feedback would be accompanied by a certain cost, and the integral (Equation 10.32) 
substantiates that hypothesis. 

By itself, the trade-off quantified by Equation 10.32 does not impose a meaningful design limitation. 
It is true that requiring a large area of sensitivity reduction over a low-frequency interval implies that an 
equally large area of sensitivity increase must be present at higher frequencies. However, it is possible to 
achieve an arbitrary large area of sensitivity increase by requiring |S(jw)| = 1+ 8, Vw € (1, w2), where 
8 can be chosen arbitrarily small and the interval (« 1, w2) is adjusted to be sufficiently large. 

The analysis in the preceding paragraph ignores the effect of limitations upon system bandwidth that 
are always present in a practical design. For example, it is almost always necessary to decrease open-loop 
gain at high frequencies to maintain stability robustness against large modeling errors due to unmodeled 
dynamics. Small open-loop gain is also required to prevent sensor noise from appearing at the system 
output. Finally, requiring open-loop gain to be large at a frequency for which plant gain is small may lead 
to unacceptably large response of the plant input to noise and disturbances. Hence the natural bandwidth 
of the plant also imposes a limitation upon open-loop bandwidth. 

One or more of the bandwidth constraints just cited is usually present in any practical design. It is 
reasonable, therefore, to assume that open-loop gain must satisfy a frequency-dependent bound of the 
form 


1+k 
ILGw)| <e(—=) Yo 2 we (10.33) 


where € < 1/2 and k > 0. This bound imposes a constraint upon the rate at which loop gain rolls off, as 
well as the frequency at which rolloff commences and the level of gain at that frequency. 
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When a bandwidth constraint such as Equation 10.33 is imposed, it is obviously not possible to require 
the sensitivity function to exceed one over an arbitrarily large frequency interval. When Equation 10.33 
is satisfied, there is an upper bound on the area of sensitivity increase which can be present at frequencies 
greater than «,. The corresponding limitation imposed by the sensitivity integral (Equation 10.32) and 
the rolloff constraint (Equation 10.33) is expressed by the following result [28]. 


Corollary 10.1: 


Suppose, in addition to the assumptions of Theorem 10.2, that L(s) satisfies the bandwidth constraint given 
by Equation 10.33. Then the tail of the sensitivity integral must satisfy 


3€W¢ 


2k © 


= 


(10.34) 


lo) 
i log |S(jo)|do 


Ic 


The bound in Equation 10.34 implies that the sensitivity trade-off imposed by the integral constraint 
in Equation 10.32 must be accomplished primarily over a finite frequency interval. As a consequence, 
the amount by which |S(jw)| must exceed one cannot be arbitrarily small. Suppose that the sensitivity 
function is required to satisfy the upper bound 


[SGio)| <a<1, Vo < og < . (10.35) 


If the bandwidth constraint (Equation 10.33) and the sensitivity bound (Equation 10.35) are both satisfied, 
then the integral constraint (Equation 10.32) may be manipulated to show that [28] 


Np 
1 3€W¢ 
sup log |S(jw)| = > Re(pi) + we log (<) — : (10.36) 
WE (wp,00¢) c— WE dX oe 2k 


The bound in Equation 10.36 shows that increasing the area of low-frequency sensitivity reduction by 
requiring a to be very small or w¢ to be very close to «,, will necessarily cause a large peak in sensitivity 
at frequencies between we and w,. Hence, the integral constraint (Equation 10.32) together with the 
bandwidth constraint (Equation 10.33) imposes a trade-off between sensitivity reduction and sensitivity 
increase which must be accounted for in design. 

It may be desirable to impose a bandwidth constraint such as Equation 10.33 directly on the comple- 
mentary sensitivity function T(s) since T(s) directly expresses the feedback properties of sensor noise 
response and stability robustness, while L(s) does so only indirectly. Analogous results to those stated in 
Corollary 10.1 can be obtained in this case. 


10.4.4 Complementary Sensitivity Integral 


The complementary sensitivity function is constrained by the stability requirement for the closed-loop 
system in a manner analogous to the sensitivity function. The following result is due to Middleton and 
Goodwin [45]. One difference between the two results is that the trade-off described by the Bode sensitivity 
integral (Equation 10.32) is worsened if the system is open-loop unstable. As we shall see, the analogous 
trade-off described by the complementary sensitivity integral becomes worse if the open-loop transfer 
function contains a time delay and/or zeros in the right half-plane. 


Theorem 10.3: Complementary Sensitivity Integral 


Suppose that the open loop transfer function L(s) is given by the product of a rational transfer function and 
a delay element, 
LG=1@)e"; (10.37) 
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FIGURE 10.7 Areas of complementary sensitivity increase (log | T(ja)| < 0) and complementary sensitivity decrease 
(log |T(jo)| > 0) for L(s) = (s+ 1)/s?. 


where L(s) is assumed to be rational with right-half-plane zeros {zj:i=1,...,Nz} (with multiple zeros 
included according to their multiplicity). If L(s) has at least one pole at the origin (i.e., one integrator), and 
if the associated feedback system is stable, then the complementary sensitivity function must satisfy 


ca dw = 1 1 1 
[ REO g (=) +3e- Say (10.38) 
where Ky is the velocity constant of the system, 
K,= lim sL(s). (10.39) 


The complementary sensitivity integral (Equation 10.38) has a similar interpretation to the Bode 
sensitivity integral. Recall that the complementary sensitivity function characterizes the response of the 
system to sensor noise and the robustness of the system to high-frequency model errors. Theorem 10.3 
states that if the open-loop transfer function is minimum phase (i.e., it has no right-half-plane zeros, and 
no delay) and is a Type II system, then the area of amplified sensor noise response must equal the area 
of attenuated sensor noise response. In the case of the complementary sensitivity function, the areas are 
computed with respect to an inverse frequency scale (see Figure 10.7). The presence of nonminimum 
phase zeros or delays worsens the trade-off (i.e., increases the required area of noise amplification). For 
Type I systems, the trade-off is improved by the term involving the velocity constant on the right-hand 
side of Equation 10.39. 

As for the sensitivity integral, the complementary sensitivity integral does not imply that the peak 
in the complementary sensitivity transfer function must be large. It is possible to accommodate the 
required increase in the magnitude of the complementary sensitivity function by allowing it to be only 
slightly greater than one for frequencies close to zero (since the area is computed with respect to inverse 
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frequency). However, when combined with tracking requirements imposed at low frequencies (analogous 
to the rolloff requirement in Equation 10.33), the integral constraint in Equation 10.38 can be used to 
develop a lower bound on the peak of the complementary sensitivity function. 

Assume that the open-loop transfer function satisfies 


, Wp 1+k 
|L(jm)| = 8 (—) » Vo < Wp, (10.40) 


where § > 2 and k > 0. This bound imposes a constraint upon the tracking performance of the system. 

When a performance constraint such as Equation 10.40 is imposed, it is obviously not possible to require 
the complementary sensitivity function to exceed one over an arbitrarily large inverse frequency interval. 
When Equation 10.40 is satisfied, there is an upper bound on the area of complementary sensitivity 
increase which can be present at frequencies less than wp. The corresponding limitation imposed by 
the complementary sensitivity integral (Equation 10.38) and the rolloff constraint (Equation 10.40) is 
expressed by the following result. 


Corollary 10.2: 


Suppose, in addition to the assumptions of Theorem 10.3, that L(s) satisfies the performance constraint 
given by Equation 10.40. Then the low-frequency tail of the complementary sensitivity integral must satisfy 


Mp dw 
| i log |T(jo)| 
0 60) 


The bound in Equation 10.41 implies that the complementary sensitivity trade-off imposed by the integral 
constraint in Equation 10.38 must be accomplished primarily over a finite inverse frequency interval. As 
a consequence, the amount by which | T(jw)| must exceed one cannot be arbitrarily small. 

Suppose that the complementary sensitivity function is required to satisfy the upper bound 


3 
2k8Mp 


= 


(10.41) 


IT(j@)|<a<1, Vo > wp > ap. (10.42) 


If the performance constraint given by Equation 10.40 and the complementary sensitivity bound in 
Equation 10.42 are both satisfied, then the integral constraint (Equation 10.38) may be manipulated to 
show that 


Nz 
1 1 
sup logit = = {= ) Re (2 


@E(Wp,Mn) 


log i 3 r T 
Ky'y. (10.43 
) F pn 2kbmp 2 Peg ( 


Wp Wh 


The bound in Equation 10.43 shows that increasing the area of high-frequency complementary sensi- 
tivity reduction by requiring a to be very small or w,, to be very close to wp will necessarily cause a large 
peak in sensitivity at frequencies between wp, and wy. Hence, the integral constraint (Equation 10.38) 
together with the performance constraint (Equation 10.40) imposes a trade-off between complementary 
sensitivity reduction and complementary sensitivity increase which must be accounted for in design. 


10.5 Limitations Imposed by Right-Half-Plane Poles and Zeros 


10.5.1 Introduction 


As discussed in Section 10.2, design specifications are often stated in terms of frequency-dependent 
bounds on the magnitude of closed-loop transfer functions. It has long been known that control system 
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design is more difficult for nonminimum phase or unstable systems. The sensitivity and complementary 
sensitivity integrals presented in Section 10.4 indicated that nonminimum phase zeros and unstable poles 
could worsen the individual design trade-offs. In fact, right-half-plane poles and zeros impose additional 
constraints upon the control system design. This section examines these limitations in detail. 


10.5.2 Limitations for Nonminimum Phase Systems 


Suppose that the plant possesses zeros in the open right-half-plane. Examples of such systems abound, 
including the inverted pendulum and cart [56], rear steering bicycle [4], fuel cells [63], acoustic ducts [54], 
and continuously variable transmission [41]. Then the internal stability requirement dictates that these 
zeros also appear, with at least the same multiplicity, in the open-loop transfer function L(s) = P(s)C(s). 
Let the set of all open right-half-plane zeros of L(s) (including any present in the compensator) be 
denoted by 

{zj:i=1,...,Nz}. (10.44) 


Defining the Blaschke product (all-pass filter) 


B.(s) =| | 2 = (10.45) 


we can factor the open-loop transfer function into the form 
L(s) = Lm(s)Bz(s), (10.46) 


where Lj;(s) has no zeros in the open right-half-plane. Note that 


IL(j@)| = |LnmGo)|, Vo, (10.47) 
and, in the limit as w > ov, 
Z— jo ° 
ipo 180% (10.48) 
Zi +jJw 


These facts show that open right-half-plane zeros contribute additional phase lag without changing the 
gain of the system (hence the term “nonminimum phase zero”). The effect that this additional lag has 
upon feedback properties can best be illustrated using a simple example. 

Consider the nonminimum phase plant P(s) = G + ran and its minimum phase counterpart P;,(s) = 
1/s+1. Figure 10.8 shows that the additional phase lag contributed by the zero at s=1 causes the 
Nyquist plot to penetrate the unit circle and the sensitivity to be larger than one. Experiments with 
various compensation schemes reveal that using large loop gain over some frequency range to obtain 
small sensitivity in that range tends to cause sensitivity to be large at other frequencies. 

Assume that the open-loop transfer function can be factored as 


L(s) = Lo(s)Bz(s)B, '(s)e™, (10.49) 
where t > 0 represents a possible time delay, Lo(s) is a proper rational function with no poles or zeros in the 


open right plane, and B,(s) is the Blaschke product (Equation 10.45) containing the open right-half-plane 
zeros of the plant plus those of the compensator. The Blaschke product 


By(s) =] | =— (10.50) 


contains all the poles of both plant and compensator in the open right-half-plane, again counted 
according to multiplicity. We emphasize once more that internal stability requirements dictate that 
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FIGURE 10.8 Additional phase lag contributed by a nonminimum phase zero. 


all right-half-plane poles and zeros of the plant must appear with at least the same multiplicity in L(s) and 
hence cannot be canceled by right-half-plane zeros and poles of the compensator. 

One constraint that right-half-plane zeros impose upon the sensitivity function is immediately obvious 
from the definition S(jw) = 1/(1 + L(jw)). Suppose that L(s) has a zero at s = z. It follows that 


S(z) =1. (10.51) 
Poles of L(s) also constrain the sensitivity function. If L(s) has a pole at s = p, then 
S(p) = 0. (10.52) 


From Equations 10.51 and 10.52, it is clear that if the plant (and thus L(s)) has zeros or poles at points 
of the open right-half-plane, then the value of the sensitivity function is constrained at those points. 
Naturally, the value of sensitivity along the jw-axis, where the design specifications are imposed and the 
conjectured trade-off must take place, is of more concern. 


Theorem 10.4: 
Suppose that the open-loop transfer function L(s) has a zero, z=x+jy, with x >0. Assume that the 
associated feedback system is stable. Then the sensitivity function must satisfy 


(10.53) 


> 


lo) 
i log |S(jo)|W(z, w)dw = m log B52) 
0 
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where W(z, w) is a weighting function. For a real zero, z = x, 


2x 


— 10.54 
x2 +? ( ) 


W(x, 0) = 


and, for a complex zero, z =x + jy, 


x nf x 
ey ge hOhew)e 


W(z, w) = (10.55) 


A number of remarks about this theorem are in order. First, as discussed in [28], the integral relations 
are valid even if S(s) has zeros (or poles) on the jw-axis. Second, a zero z with multiplicity m > 1 imposes 
additional interpolation constraints on the first m— 1 derivatives of log S(s) evaluated at the zero. These 
interpolation constraints also have equivalent statements as integral relations [28]. 

We now show that Equation 10.53 imposes a sensitivity trade-off. To see this, note first that the 
weighting function satisfies W(z,w) > 0,Vw, and that the Blaschke product satisfies log |B, ‘(z)| = 0. 
Using these facts, it follows easily from Equation 10.53 that requiring sensitivity reduction (log |S(jw)| < 
0) over some frequency range implies that there must be sensitivity amplification (log |S(jw)| > 0) at other 
frequencies. Hence, if the plant is nonminimum phase, one cannot use feedback to obtain the benefits of 
sensitivity reduction over one frequency range unless one is willing to pay the attendant price in terms of 
increased sensitivity elsewhere. 

Theorem 10.4 verifies the conjecture that a sensitivity trade-off is present whenever a system is non- 
minimum phase. Recall it was also conjectured that the severity of the trade-off is a function of the phase 
lag contributed by the zero at frequencies for which sensitivity reduction is desired. This conjecture can 
be verified by using the form of the weighting function W(z, w) defined in Equations 10.54 and 10.55. 

Consider first the case of a real zero z = x. Equation 10.46 shows that, as a function of frequency, the 
additional phase lag contributed by this zero is 


A(x, 0) = Z—*. (10.56) 
xX+jW 
Noting that 
d0(x, w) —2x 
= ; 10.57 
dw x? + w? ( ) 
it follows that the weighting function in Equation 10.54 satisfies 
d(x, 
Wiis ie BO). (10.58) 
do 


Hence the weighting function appearing in the sensitivity constraint is equal to (minus) the rate at which 
the phase lag due to the zero increases with frequency. 

One can use the weighting function (Equation 10.58) to compute the weighted length of a frequency 
interval. Note that sensitivity reduction is typically required over a low-frequency interval Q = (0, 1) 
and that the weighted length of such an interval equals 


1 
W(x, Q) = W(x, w)dw 
0 


= —0(x, 1). (10.59) 


Hence, the weighted length of the interval is equal to (minus) the phase lag contributed by the zero at the 
upper endpoint of the interval. It follows that, as w; —> 00, the weighted length of the jw-axis equals 1. 
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For a complex zero, the weighting function (Equation 10.55) is equal to minus the average of the 
additional phase lag contributed by the zero and by its complex conjugate: 


1 (d0(z,w)  d0(Z, w) 
W(z,o) = ; 10.60 
Gyo) 2 ( doy t doy ) ( 
Hence, the weighted length of the frequency interval Q = (0, w1) is 
1 
W(x, 0) = —5 (0(z, w1) + 0G, o1)) - (10.61) 


As we have already remarked, the integral constraint in Equation 10.53 implies that a trade-off exists 
between sensitivity reduction and sensitivity increase in different frequency ranges. An interesting inter- 
pretation of this trade-off is available using the weighting function. Suppose first that L(s) has no poles in 
the open right-half-plane. Then the integral constraint is 


/ ‘ log |S(jw)| W(x, ) dw = 0. (10.62) 
0 


Equation 10.62 states that the weighted area of sensitivity increase must equal the weighted area of 
sensitivity reduction. Since the weighted length of the jw-axis is finite, it follows that the amount by which 
sensitivity must exceed one at higher frequencies cannot be made arbitrarily small. 

If the open-loop system has poles in the open-right-half-plane, then the weighted area of sensitivity 
increase must exceed that of sensitivity reduction. In particular, 


4 (10.63) 


Np = 
log [3,1 = Y “log E = 
i=1 ’ 
The right-hand side of Equation 10.63 is always greater than zero, and becomes large whenever the zero 
z approaches the value of one of the unstable poles p;. It follows (unsurprisingly) that systems with 
approximate pole-zero cancellations in the open right half-plane will necessarily have poor sensitivity 
properties. 

We can use the integral constraint in Equation 10.53 to obtain some simple lower bounds on the size of 
the peak in sensitivity accompanying a given level of sensitivity reduction over a low-frequency interval. 
Bounds of this type were first discussed by Francis and Zames [26, Theorem 3]. The results presented 
here will show how the relative location of the zero to the interval of sensitivity reduction influences the 
size of the peak in sensitivity outside that interval. 

Suppose that the sensitivity function is required to satisfy the upper bound 


|SGjw)| <a <1, (10.64) 


where Q = (0, ) is a low-frequency interval of interest. Define the infinity norm of the sensitivity 
function: 
[ISlloo = sup |S(jo)]. (10.65) 
wo>0 
Assuming that the upper bound in Equation 10.64 is satisfied, the integral constraint in Equation 10.53 
can be used to compute a lower bound on ||S||,o for each nonminimum phase zero of L(s). 


Corollary 10.3: 


Suppose that the conditions in Theorem 10.4 are satisfied and that the sensitivity function is bounded 
as in Equation 10.64. Then the following lower bound must be satisfied at each nonminimum phase 
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zero of L(s): 


W(z,Q) 


mt—W(z,Q) 
ISlloo > (<) 
Qa 


The bound in Equation 10.66 shows that if sensitivity is required to be very small over the interval 


1 
n—W(z,Q) 


B, (z) 


(10.66) 


(0, w 1), then there necessarily exists a large peak in sensitivity outside this interval. Furthermore, the 
smallest possible size of this peak will become larger if the open-loop system has unstable poles near any 
zero. 

The size of the sensitivity peak also depends upon the location of the interval (0, w1) relative to the 
zero. Assume for simplicity that the system is open-loop unstable and the zero is real. Then 


1 W(x,Q) 
m—W(x,Q) 
ISlloo 2 (=) . (10.67) 


Recall that the weighted length of the interval 2 = (0, w ) is equal to (minus) the phase lag contributed 
by the zero at the upper endpoint of that interval. Since the zero eventually contributes 180° phase lag, it 
follows that as w; — 00, W(x, Q) > a. Thus the exponent in Equation 10.67 becomes unbounded and, 
since a < 1, so does the peak in sensitivity. To summarize, requiring sensitivity to be small throughout a 
frequency range extending into the region where the nonminimum phase zero contributes a significant 
amount of phase lag implies that there will necessarily exist a large peak in sensitivity at higher frequencies. 
On the other hand, if the zero is located so that it contributes only a negligible amount of phase lag at 
frequencies for which sensitivity reduction is desired, then it does not impose a serious limitation upon 
sensitivity properties of the system. Analogous results hold, with appropriate modifications, for a complex 
zero. 

Suppose now that the open-loop system has poles in the open right-half-plane. It is interesting to note 
that, in this case, the bound (Equation 10.66) implies the existence of a peak in sensitivity even if no 
sensitivity reduction is present! 

Recall next the approximation in Equation 10.26 which shows that small sensitivity can be obtained 
only by requiring open-loop gain to be large. It is easy to show that |S(jw)| < a < 1 implies that |L(jw)| > 
(1/a) — 1. The inequality in Equation 10.67 implies that, to prevent poor feedback properties, open-loop 
gain should not be large over a frequency interval extending into the region for which a nonminimum 
phase zero contributes significant phase lag. This observation substantiates a classical design rule of 
thumb: loop gain must be rolled off before the phase lag contributed by the zero becomes significant. 
However, if one is willing and able to adopt some nonstandard design strategies (such as having multiple 
gain crossover frequencies) then [37] it is possible to manipulate the design trade-off imposed by a 
nonminimum phase zero to obtain some benefits of large loop gain at higher frequencies. One drawback 
of these strategies is that loop gain must be small, and hence the benefits of feedback must be lost over an 
intermediate frequency range. 


10.5.3 Limitations for Unstable Systems 


We shall show in this section that unstable poles impose constraints upon the complementary sensitivity 
function which, loosely speaking, are dual to those imposed upon the sensitivity function by nonminimum 
phase zeros. That such constraints exist might be conjectured from the existence of the interpolation 
constraint in Equation 10.52 and the algebraic identity in Equation 10.21. Together, these equations show 
that if L(s) has a pole s = p, then the complementary sensitivity function satisfies 


T(p)=1. (10.68) 
Furthermore, if L(s) has a zero at s = z, then 


T(z) =0. (10.69) 
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The previous results for the sensitivity function, together with the fact that T(s) is constrained to equal 
one at open right-half-plane poles of L(s), suggests that similar constraints might exist for |T(jw)| due to 
the presence of such poles. It is also possible to motivate the presence of the integral constraint on |T(jw)| 
using an argument based upon the inverse Nyquist plot [53] and the fact that |T(jw)| > 1 whenever 
L~1!(jo) is inside the unit circle centered at the critical point. 

As in Section 10.5.2, it is assumed that L(s) has the form given in Equation 10.49. The following theorem 
states the integral constraint on the complementary sensitivity function due to unstable poles. 


Theorem 10.5: 


Suppose that the open-loop transfer function has a pole, p= x + jy, with x > 0. Assume that the associated 
feedback system is stable. Then the complementary sensitivity function must satisfy 


/ x log |T(j@)|W (x, @)dw = x log |By'(p)| + xxt, (10.70) 
0 


where W(p, w) is a weighting function. For a real pole, p = x, 


2x 


W(x,0) = =———, 
(60) = 5 


(10.71) 


and, for a complex pole, p = x + jy, 
x x 


WP = Typ Pty For 


(10.72) 


Remarks analogous to those following Theorem 10.4 apply to this result also. The integral relations 
are valid even if T(s) has zeros on the jw-axis, and there are additional constraints on the derivative of 
log T(s) at poles with multiplicity greater than one. 

The integral constraint in Equation 10.70 shows that there exists a trade-off between sensor noise 
response properties in different frequency ranges whenever the system is open-loop unstable. Since 
|T (ja)| is the reciprocal of the stability margin against multiplicative uncertainty, it follows that a trade-off 
between stability robustness properties in different frequency ranges also exists. Using analysis methods 
similar to those in the preceding section, one can derive a lower bound on the peak in the complementary 
sensitivity function present whenever |T(jw)| is required to be small over some frequency interval. One 
difference is that |T(jw)| is generally required to be small over a high, rather than a low, frequency 
range. 

It is interesting that time delays worsen the trade-off upon sensor noise reduction imposed by unstable 
poles. This is plausible for the following reason. Use of feedback around an open-loop unstable system is 
necessary to achieve stability. Time delays, as well as nonminimum phase zeros, impede the processing of 
information around a feedback loop. Hence, it is reasonable to expect that design trade-offs due to unstable 
poles are exacerbated when time delays or nonminimum phase zeros are present. This interpretation 
is substantiated by the fact that the term due to the time delay in Equation 10.70 is proportional to 
the product of the length of the time delay and the distance from the unstable pole to the left half- 
plane. 


10.5.4 Summary 


Nonminimum phase or unstable systems impose additional trade-offs for control system design. Non- 
minimum phase zeros limit the frequency range over which control system performance can be achieved, 
while unstable poles require active control over certain frequency ranges and reduce the overall per- 
formance that can be achieved. Quantitative expressions of these trade-offs are given by the integral 
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constraints of Theorems 10.4 and 10.5. These constraints can be used together with bounds on the 
desired performance to compute approximations that provide useful insight into the design trade-offs. 


10.6 Time-Domain Integral Constraints 


10.6.1 Introduction 


One class of limitations on feedback system performance can be described as time domain integral 
constraints. These date back at least as far as classical works such as [68], which analyze various error 
constants and their effects on performance. Such constraints, for single-degree-of-freedom unity feed- 
back systems, can be described in a straightforward manner using classical control techniques and 
have been explored for example in [43,45]. They are based on the unity feedback system depicted in 
Figure 10.1. 


10.6.2 Double Integrators 


As an initial case, suppose that we have a feedback loop that incorporates two integrators. For example, 
this may arise from a situation where the plant includes an integrator and we also specify a controller that 
incorporates integral action. In this case, it follows that the open-loop transfer function can be written as 
L(s) = 5 Ls), where L(0) 4 0. Then consider the error response to a unit step change in the reference, 


R(s)= 2 
2 
BG) = SR) = ( : )e=( : ) (10.73) 


S? 
1+L(s)) s \st+L(s)/ s’ 


Assuming closed-loop stability, it follows that s = 0 is in the region of convergence of E(s) and E(0) = 
S(0)R(0) = 0, that is, 
[e,3) 
E(0) = / e(t) dt = 0. (10.74) 


0 


Equation 10.74 can be viewed as an equal area criterion on the step response. This in turn implies that 
either the error signal is zero for all time, or it must alternate signs. Therefore, since the error must be 
negative for some times, there must be overshoot in the step response, since y(t) = 1 — e(t) will exceed 
1 whenever e(t) is negative. Figure 10.9 illustrates an example response for a system containing a double 
integrator. 

We conclude from this motivating example that a stable unity feedback system with a double integrator 
displays unavoidable overshoot in the step response. More general versions of the integral constraint 
(Equation 10.74) can be derived to quantify the effect of open-loop CRHP poles and zeros on the closed- 
loop step response, as we discuss next. 


10.6.3 Right-Half-Plane Poles and Zeros 


We have seen in Section 10.5 that open-loop right-half-plane poles and zeros impose interpolation 
constraints on the sensitivity and complementary sensitivity functions. Namely, if L(s) has a CRHP 
zero at s =z, then for any internally stable closed-loop system the interpolation constraints given in 
Equations 10.51 and 10.69 hold. On the other hand, if L(s) has a CRHP pole at s = p, then Equations 10.52 
and 10.68 hold. 

Recall that S(s) and T(s) govern the output and error response of the system to exogenous inputs 
through Equations 10.4 and 10.5. Closed-loop stability implies that CRHP poles and zeros are in the region 
of convergence of the Laplace transform, and it thus follows from Equation 10.52 that E(p) = S(p)R(p) = 0, 
and from Equation 10.69 that Y(z) = T(z)R(z) = 0. These constraints are expressed in integral form in 
the following theorem. 
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FIGURE 10.9 Example closed loop error step response for a double integrator open loop transfer function. 


Theorem 10.6: 


Consider any closed-loop stable linear time-invariant SISO system as illustrated in Figure 10.1 with zero 
initial conditions and a step reference signal. Then, for any CRHP pole p of P(s), the error signal e(t) satisfies 


[e.e) 


fermen dt = 0. (10.75) 
0 


Furthermore, for any CRHP zero z of P(s), the output signal y(t) satisfies 


[oe 


[eww dt =0. (10.76) 
0 


The integral constraints in Equations 10.75 and 10.76 are weighted equal area criteria on the step 
responses e(f) and y(t). Some direct consequences of these results follow as trade-offs in the step response 
specifications (see Figure 10.10) in the case of real CRHP poles and zeros (see [43,45] for further details): 


¢ The closed-loop unity feedback step response of a plant with a real RHP pole must overshoot, that 
is, the error signal, e(t), must change sign. If we take a slightly unusual definition of rise time 


t, = max{t, : y(t)<t/t, Vt<t}, 


and if we define the overshoot as the maximum amount by which the output exceeds the reference, 
Yos = Max;>o0{y(t) — 1}, then for a real open-loop pole at s = p > 0, the overshoot must satisfy the 
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FIGURE 10.10 Step response specifications. 


lower bound 
Bis 
a 
¢ The closed-loop step response of a plant with a real RHP zero must undershoot, that is, there must 
exist times for which the output is negative. If we take the non standard definition of settling time 


Vos = (10.77) 


ts =min{t,: y(t)>0.9 Vt> ts}, 
t.>0 


then for any system with a real open-loop zero at s = z > 0, the undershoot must satisfy the lower 


bound 
0.9 


ets — 1 

The above time domain integrals provide a number of insights into fundamental limits on the behavior 
of stable closed-loop systems with real RHP open-loop poles or zeros. The effects of slow OLHP open- 
loop poles on transient performance have been examined in [46]. Trade-offs in the step response due to 
jo-axis zeros have been studied in [33]. Extensions to multivariable systems has been pursued in [40], in 
which directional properties of multivariable systems may permit step response trade-offs that include 
possible transients in other loops. 


Vus = min{ y(t)} = (10.78) 


10.7 Further Extensions and Comments 


A range of extensions to the basic trade-offs and limitations discussed previously are available. In this 
section, we briefly review some of the main points and key references for a selection of these results. We 
conclude the section with a brief discussion of cases in which tracking performance limitations may be 
alleviated. 


10.7.1 Limitations in Discrete-Time and Sampled-Data Control 


Integral constraints imposed by feedback stability on the sensitivity and complementary sensitivity func- 
tions also apply to discrete-time systems. The first discrete-time extensions of the Bode integral for the 
sensitivity and complementary sensitivity functions were obtained by Sung and Hara [64,65]. A unified 
formulation of continuous and discrete-time results appeared in [43] and [45]. These results show that 
the design trade-offs arising from these integral constraints in continuous-time systems carry over to 
discrete-time systems with analogous interpretations. 
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FIGURE 10.11 Linear time-invariant discrete-time feedback system. 


We present here the discrete-time Bode sensitivity integral as given in [64]. Consider the discrete- 
time feedback system shown in Figure 10.11, where all signals are now functions of the discrete-time 
variable k = 0, 1, 2,..., and the plant and controller are represented by the discrete-time rational transfer 
functions Pg(z) and Cq(z). 

The discrete-time open-loop, sensitivity, and complementary sensitivity functions are defined analo- 
gously to their continuous-time counterparts: 


TC ee 


La(z) = Pa(z) Ca(z),  Sa(z) = = eae 


L+Lq(z)’ 


Suppose that the open-loop transfer function Lq(s) has no pole-zero cancellations in D* = {z : |z| > 1}. 
Then the discrete-time closed-loop system is stable if Sy(z) and Ty(z) have no poles in the open unit disk 
D= {z: |z| < 1}. 


Theorem 10.7: Bode Discrete-Time Sensitivity Integral 


If the loop transfer function Lq(z) is strictly proper and the closed-loop system shown in Figure 10.11 is 
stable, then 

a a = 

-{ log |Sa(e”)| do =)“ log |i, (10.79) 


i=1 


where {; € D°,i=1,... , Np} are the unstable poles of La(z). 


As in the continuous-time case, the Bode discrete-time sensitivity integral (Equation 10.79) shows that 
there also exists a balance of areas of sensitivity reduction and amplification for discrete-time systems. A 
significant difference with the continuous-time case is that in the Bode discrete-time sensitivity integral 
(Equation 10.79) integration is performed over a finite interval, and thus the balance of areas of sensitivity 
reduction and amplification directly implies design trade-offs, even if no bandwidth constraints are 
imposed on the discrete-time system. 

One should note that if the discrete plant corresponds to the linear time-invariant discretization of an 
analog plant to be controlled digitally through periodic sampling and hold devices, the analysis of design 
trade-offs and limitations based on the study of discrete-time sensitivity functions may not capture the real 
(continuous-time) behavior. Indeed, it is known, for example, that the sampled zeros of a discretized plant 
can be manipulated by appropriate choice of the hold device [24, Chapter 10]. Hence, the discretization 
of a nonminimum phase analog plant may be rendered minimum phase, consequently relaxing design 
limitations in the sampled response. However, design limitations and design trade-offs remain in the 
intersample response independently of the type of hold device used, as shown in [29]. Intersample 
behavior must thus be taken into account to detect potential difficulties in the continuous-time response. 
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FIGURE 10.12 Feedback control over a communication channel. 


10.7.2 Limitations in Control with Communication Constraints 


The increased application of communications and networking technology in the new generation of control 
systems has prompted many recent studies of limitations in control schemes in which the feedback loop 
is closed over a communication channel (see, e.g., the papers in [2] and the references therein). While a 
comprehensive theory of limitations in control over communication channels is yet unavailable, several 
important questions have been answered for simplified channel models (see, e.g., [11,50]). 

A simple scheme of a (discrete-time) control system with feedback over a communication channel is 
illustrated in Figure 10.12. Here, the block C; encapsulates processes such as encoding, sensor filtering, and 
control computations to generate the signal 1; sent over the communication channel. The communication 
channel may include quantization effects, delay, noise, data loss, and data-rate contraints. The block C, 
encapsulates processes such as decoding and actuator signal processing based on the received signal wx. 

This section presents a generalization of the discrete-time Bode Sensitivity Integral seen in 
Section 10.7.1 to systems with feedback over a noisy communication channel with limited capacity. 
This result links a fundamental limitation to feedback performance to a fundamental limitation in com- 
munications by using the information theoretic concepts developed by Shannon [60]. 

For an additive white Gaussian noise channnel (AWGN) model, the received signal w; in Figure 10.12 
is given as 


Wk = Vk + Nk, (10.80) 


where nx is a white Gaussian signal with variance V independent of the transmitted signal v,. The 
transmitted signal v; is subject to a channel input power constraint E{vz} < P. The (information) capacity 
C of the channel is given by the famous Shannon formula [60] 


1 P ; 
C= > log, (1 + =) bits/sample. (10.81) 


Given a source that produces information at rate R, Shannon showed that it is possible to communicate 
this information reliably (i.e., with arbitrarily small probability of error) over a channel with capacity 
C > R. On the other hand, if R > C, then reliable communication is not possible. 

Suppose that a feedback system uses information transmitted over a communication channel in the 
feedback loop, as depicted in Figure 10.12. Then it is natural to ask how the capacity of the channel 
limits properties of the feedback system. For example, it is of interest to determine the minimum channel 
capacity required to stabilize an unstable discrete-time plant. The authors of [50,66] show that stabilization 


is possible if and only if the data rate of the channel is greater than eae log |;|, where the ; are 
the unstable plant eigenvalues. Similarly, it is shown in [11] that an unstable discrete-time plant with 
relative degree equal to one and no zeros outside the open unit disk may be stabilized over a Gaussian 
communication channel if and only if the channel capacity (Equation 10.81) satisfies the same lower 
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bound identified in [50]: 
C= > / log |dil. (10.82) 


In addition to stabilization, feedback may also be used in a control design for the purpose of disturbance 
attenuation. We have seen that the Bode sensitivity integral limits the ability to attenuate disturbances for 
the discrete-time feedback system of Figure 10.11. A generalized version of the Bode sensitivity integral 
derived by Martins and Dahleh [42] shows that the disturbance attenuation properties of a feedback 
system with a communication channel in the feedback loop, as depicted in Figure 10.12, are limited 
by the capacity of the channel. To state this result, assume in Figure 10.12 that the blocks C; and C, 
are (possibly nonlinear and time-varying) causal operators, and let the exogenous disturbance d;, be a 
stationary Gaussian moving average process with a nontrivial power spectral density ®g(w). Assume also 
that the feedback system is mean-square stable, and that the control input sequence u; is asymptotically 
stationary and has a well-defined power spectral density ®,,(w). Then a sensitivity function can be defined 


as [42, Definition 3.2] 
S(o) = VO, (@)/ O40). (10.83) 
The following result is found in [42, Theorem 6.3]. 


Theorem 10.8: 


If the transfer function Pg(z) is strictly proper and the closed-loop system shown in Figure 10.12 is stable, 
then 
LF a 
_ [ min{0, log S;(w)} da = Y log Idi] —C, (10.84) 
nae k=1 


where {; € D°,i=1,... , Np} are the unstable poles of Pa(z). 


Theorem 10.8 shows that if the capacity of the Gaussian channel is equal to the minimum required for 
stabilization, then it is not possible to achieve S;(w) < 1 at any frequency, and thus disturbance attenuation 
is not possible. If disturbance attenuation is required in a feedback design, it is thus necessary that the 
channel capacity exceed that required for stabilization. 

Although we have presented Theorem 10.8 in the context of a Gaussian channel, in fact, the result 
holds for any channel, and the reader is referred to Martins and Dahleh [42] for further discussion. 


10.7.3. Cheap Control and Achievable Hy Performance 


An alternate approach to considering some performance limitations has been to consider so-called cheap 
control problems, wherein an output performance is optimized, while (for the sake of analysis) ignoring 
any cost associated with the control signal. For example, it was shown in [52] that for the system as shown 
in Figure 10.1, and with a unit step reference, the minimum achievable quadratic error performance for 
any stabilizing control, even given full state measurement and a nonlinear controller, must satisfy the 
so-called cheap control performance limitation: 


lo) 
i. e*(t)dt > 2 dS eae (10.85) 
0 zj¢ORHP 


A series of works such as [17] considered a similar problem, but in an output feedback situation where 
it was shown that in the case of linear time-invariant control, the inequality in Equation 10.85 is strict if 
and only if the plant includes both ORHP poles and zeros. 
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The quantification of performance limitations via cheap control (time domain) has been linked to the 
Bode sensitivity and complementary sensitivity integrals (frequency domain) by Seron et al. [57] and 
Middleton and Braslavsky [44], the former of which also extends the analysis to nonlinear systems. A key 
observation in that paper is that the best achievable performance, as the cost associated with the control 
signal tends to zero, is determined by the least amount of “energy” (in the Lz sense) required to stabilize 
the unstable zero dynamics of the plant, which is nonzero if the plant is nonminimum phase. 

Limitations to optimal cheap control performance have been studied also in linear filtering problems 
[9,39], and decoupling problems in multivariable systems [12]. 


10.7.4 MIMO Systems 


Multivariable feedback systems incorporate a number of extra challenges compared to SISO loops. In 
particular, directional properties of transfer functions may play an important role. In addition, the range 
of transfer functions that are important to overall performance is larger, since the relevant open-loop 
transfer function matrices, Lo(s) = P(s)C(s) and L;(s) = C(s)P(s), are identical only in special cases. In the 
multiple input multiple output (MIMO) case, the output sensitivity, S,(s), and complementary sensitivity, 
To(s), (both of which take values in C”*™ where m is the number of outputs) are defined as 


So(s) = (I+ Lo(s))~* 


(10.86) 
To(s) = (I+ Lo(s))~*Lo(s). 


Several classes of results have been obtained for multivariable systems, and we briefly review some of 
the key points below. For simplicity of exposition, we restrict attention to (1) loops referred to the output; 
(2) cases where the loop transfer function and its inverse are generically full rank; and (3) cases where the 
open-loop poles and zeros are disjoint. In particular, we assume: 


det(L,(s)) =0 only for isolated values of s € C 
det(L>! (s))=0_ only for isolated values of s € C (10.87) 
{det(L,(z)) =0} => {det(L; '(z)) # o} for any z EC. 


10.7.4.1 “Average” Sensitivity Integral 


Consider the output sensitivity and output complementary sensitivity functions defined in Equation 10.86. 
There are various measures of interest for a transfer function matrix, but one that is particularly amenable 
to performance limitations analysis is the determinant. In particular, as shown in [28], 


Np 
a log |det(So(jo))| dw = x ) > Re(p)). (10.88) 
0 


i=1 


If we let o;(-) denote the ith singular value of a matrix, Equation 10.88 can be rewritten as 


fp dre 5j(So(j@))) downy Rep) 


Therefore, this generalization of the Bode Sensitivity Integral (Equation 10.32) gives a measure of the 
overall performance. However, it lacks detail about the worst-case or individual loop performances and 
therefore several extensions of this analysis have been considered. 
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10.7.4.2 Sensitivity Integral Inequalities 


One form of generalizations that provides some alternate information to the “Average” performance 
results of Section 10.7.4.1 examines the behavior of the maximum singular value (i-e., the induced 2-norm) 
of the output sensitivity matrix. Indeed, it can be shown (see, e.g., [8,15]) that 


/ log ||So(jo) | dw >m max Re(pj). (10.89) 
0 i=1..Np 


Inequality in Equation 10.89 arises from the directional properties of the multivariable system (see, 
e.g., Definition 10.1), which we now turn to study in more detail. 


10.7.4.3 Direction-Based Analysis 


Directional properties of vector signals and matrix transfer functions play an important role in multi- 
variable systems. The appropriate MIMO generalization of a zero* of the loop transfer function matrix, 
L,(s), is as an isolated value of s for which L,(s) drops rank. Under Equation 10.87 this is equivalent 
to isolated values of s for which det(L,(s)) = 0. Similarly, a pole of L,(s) is an isolated value of s for 
which det(L;!(s)) = 0. This leads to the following definition of MIMO zeros and poles and their input 


directions". 


Definition 10.1: Multivariable Poles and Zeros 


1. Subject to Equation 10.87, we say Lo(s) has a zero at s = z with input direction d, if 
Lo(z)dz = 0. 

2. Subject to Equation 10.87, we say Lo(s) has a pole at s = p with input direction dy if 
Lz \(p)dp = 0. 


It then follows that we have interpolation constraints on the output sensitivity and complementary 
sensitivity functions as follows. 


Lemma 10.1: Multivariable Interpolation Constraints 


Subject to Equation 10.87, we have the following: 


1. Ifs=z isa zero of Lo(s) with input direction dz, then 


So(z)dz = dz, 
T,(z)d; =0. 


* Note that in fact there are a number of different definitions of MIMO zeros. In this case, we are considering MIMO 
transmission zeros. 
+ Similar definitions and results apply to output directions, which we omit for brevity. 


Trade-Offs and Limitations in Feedback Systems 10-31 


2. Ifs=p isa pole of L(s) with input direction dp, then 


So(p)dp = 0, 
To(p)dp = dp. 


Using Lemma 10.1 we can generalize earlier SISO results such as Theorems 10.4 and 10.5 as follows: 


Theorem 10.9: 


Take any appropriately dimensioned vector d. 


1. Ifs=z is a zero of Lo(s) with input direction dz, then 


lo) 
/ log |dS.(jw)dz W(z,0)dw > wlog |d" d;|. 
0 


2. Ifs=p isa pole of L,(s) with input direction dp, then 
CO 
/ log la" To(jo)dp| W(p, @)dw > x log |d? dp. 
0 


Note that the results in Theorem 10.9 are inequalities, since in general the expression for the relevant 
Blaschke products are much more complex in this case than in the SISO case (Equation 10.50). We also 
note that by taking d aligned with d, or dp as appropriate, and using unit length vectors, we can make the 
right-hand sides of the bounds in Theorem 10.9 equal to zero. 


10.7.4.4 Other MIMO Results 


There are a wide range of multivariable feedback results available, and this section has concentrated on a 
few specific forms of these results. A summary and analysis of many of the available results is contained 
in [56]. The results of [17] considered achievable Hz performance of multivariable systems for example. 
The authors of [30] consider systems in which the performance variable is not identical to the measured 
variable and the disturbance and actuator affect the system through different dynamics. A range of other 
results are included in the special issue [16]. 


10.7.5 Time-Varying and Nonlinear Systems 


The theory of fundamental feedback limitations for general time-varying and nonlinear systems is much 
less developed than that for linear time-invariant systems. One of the main challenges in extending notions 
such as the Bode integral is that, in contrast to the linear time-invariant case, it is in general impossible 
to obtain a complete characterization of the system action as an operator via a simple transformation to 
the frequency domain. A number of significant results, however, have been obtained in the last couple of 
decades showing that some fundamental feedback limitations remain in more general classes of systems. 

One way of extending notions associated with transfer functions to nonlinear systems is to apply the 
theory of input/output (I/O) nonlinear operators on linear spaces. Using this approach, Shamma [59] 
shows that nonminimum phase dynamics impose an “area balance” constraint to the nonlinear sensitivity 
I/O operator analogous to that captured by the Bode integral for linear systems. The result shows that if 
the nonlinear plant is nonminimum phase (defined by conditions on the domain and range of the plant 
I/O operator [56, §12]), an arbitrarily small value of frequency-weighted sensitivity necessarily implies 
an arbitrarily large response to some admissible disturbance. A dual “area balance” constraint exists for 
the nonlinear complementary sensitivity I/O operator when the plant is open-loop unstable [56, $13.4]. 
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Nonlinear equivalents of the interpolation constraints (Equations 10.51 and 10.69) can also be developed, 
and applied to quantify control design trade-offs [56, $13] and [58]. 

Another approach to characterize fundamental limitations in classes of nonlinear systems arises from 
the study of cheap control problems, as in Section 10.7.3. This idea has been applied to study limitations 
in nonminimum phase, and nonright invertible strict-feedback nonlinear systems [10,57]. 

An information theoretic approach has been pursued in Iglesias [38,69] to obtain a time-domain 
interpretation of the Bode sensitivity integral and extend fundamental sensitivity limitations results 
to classes of time-varying, and nonlinear systems. Following a similar approach, Martins and Dahleh 
[42] extended the Bode sensitivity integral to control systems with feedback over capacity-constrained 
communication channels (see Section 10.7.2) that may include arbitrary time-varying and nonlinear 
components in the loop, with the only restriction of being causal. 


10.7.6 Alleviation of Tracking Performance Limitations 


From the previous discussions, it can be seen that there are a number of inherent limitations on feedback 
system performance, including tracking performance. For plants with CRHP zeros, we have seen a number 
of fundamental limitations on achievable performance. For example, the time-domain integrals based on 
CRHP zeros in Theorem 10.6 constrain the output behavior, for any reference signal, for any stabilizing 
control. In particular, a simple generalization of the time-domain integral in Theorem 10.6 shows that 
with a nonminimum phase zero at s = z and a given reference signal with Laplace transform, R(s), then 


ie e e(t) dt = R(z). (10.90) 
0 


Frequency-domain integrals such as the complementary sensitivity integral (see Section 10.4.4) and 
the Poisson sensitivity integral (Theorem 10.4) apply for any stabilizing linear time-invariant control 
scheme. In addition, there are various extensions of the cheap control results of Equation 10.85 to 
alternate reference signals. Here we wish to consider various problem settings in which it may be possible 
to alleviate the limitations on tracking performance imposed by plant CRHP zeros. 


10.7.6.1 Preview Control 


Preview control refers to a scenario wherein the reference trajectory, r(t), may be prespecified, and 
therefore, at time t, the control may use advance knowledge of future or impending reference changes, 
r(t):t € [t,t + Tpre]. In the case of infinite preview (Tpre > --00), references such as [14] showed how 
nonlinear system inversion (and therefore perfect tracking) may be performed. A more detailed analysis 
of such systems from a performance limitations perspective is discussed in [18,47]. In particular, it is 
shown from a time-domain integral perspective; from a Poisson sensitivity integral perspective; and, 
from an achievable Hy. performance perspective, that use of preview Tpre with Re{zTp;e} sufficiently 
large almost eliminates the tracking performance limitations due to a CRHP plant zero z. Of course, such 
preview alters only reference tracking performance and does not alter feedback properties such as noise 
performance, disturbance response, and stability robustness. 


10.7.6.2 Path Tracking 


Path tracking (see, e.g., [1,48]) is an alternate control paradigm in which a reference trajectory, r(t), is not 
given as a prespecified function of time. Instead, the primary tracking objective may be to ensure that at 
all times, the output is close to a prespecified path (r(6(f))), that is, that ||y(t) — r(@(t))|| is small for all 
time where 6(f) is, at least partially, free for the control designer to choose. This allows some additional 
degrees-of-freedom, and may be helpful in removing some of the limitations on tracking performance 
imposed by nonminimum phase zeros. 
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For example, if we have some freedom in the selection of r(t), it may be possible to select 6(t) in 
such a way that R(z) is small or even zero. In this case, the constraint (Equation 10.90) on the time- 
domain performance does not necessarily demand poor tracking performance. Clearly, the feedback 
noise, disturbance and robustness properties will still be affected by the usual performance trade-offs. 


10.7.6.3 Reset Controllers and Other Hybrid Structures 


Reset controllers have been shown to overcome some of the limitations inherent to linear time-invariant 
control, alleviating design trade-offs in the system time response, such as that between overshoot and rise 
time discussed in Section 10.6 [6,25]. A reset controller is a linear time-invariant system with states that 
reset to zero when the controller input is zero. This idea was first introduced by Clegg [19], who studied 
the effect of a reset integrator in a feedback loop. A reset control system may be viewed as a hybrid system, 
which incorporates both continuous dynamics and discrete events. It is possible that hybrid systems 
present a different set of performance limitations, and advantages in some problems as those discussed 
in [6,25]. However, a general theory of performance limitations for hybrid control systems remains to be 
developed. 


10.8 Summary and Further Reading 


In this chapter, we have discussed design limitations and trade-offs present in feedback design problems. 
Beginning with the pioneering work of Bode on fundamental feedback relationships, and their implica- 
tions for feedback amplifier design, a rather complete theory has been developed for linear time-invariant 
feedback systems. The core of this chapter presented the main results of this theory, as quantified by funda- 
mental relationships in the frequency domain for the sensitivity and complementary sensitivity functions. 
As discussed, these relationships arise from the basic structure of the feedback loop and the requirement 
of feedback stability, and are parameterized by the right-half-plane poles and zeros, and time delays of 
the open-loop system. These relationships have direct implications on the achievable performance and 
robustness properties of the feedback loop, and thus help a control engineer to achieve informed design 
trade-offs. 

There have been many extensions to the theory of fundamental limitations since the first version of this 
article was published. We have included a brief account of several of these topics: alternative developments 
in the time-domain, extensions to multivariable systems, time-varying and nonlinear systems, and control 
systems with feedback over a communication channel. Finally, we have also included a section discussing 
the alleviation of performance limitations in some tracking problems. 

Diverse applications of the theory of fundamental design limitations also abound. A few examples 
to illustrate the scope of applications include combustion [5], platooning [55], magnetic bearings [67], 
haptics [35], ship roll stabilization [51], and autocatalysis [13]. 

The study of performance limitations in feedback systems is at the core of feedback design, and can 
provide helpful knowledege in practical control engineering problems. On the one hand, the knowledge 
of these limitations and their associated trade-offs provides benchmarks against which different control 
structures and designs may be compared, and guides the deliberate selection of a design solution that 
achieves a reasonable compromise between conflicting design goals. On the other hand, if a reasonable 
compromise cannot be achieved for a particular system, then knowledge of design trade-offs may be used 
to modify the plant, to improve sensors and actuators, and to develop better models so that a tractable 
design problem is obtained. 

The reader who wants to know more can find material in many places. Several textbooks include 
chapters on the theory of fundamental design limitations, including Franklin et al. [27], Doyle et al. [23] 
Zhou et al. [70], Goodwin et al. [34], Skogestad and Postlethwaite [61], Astrém and Murray [3], and Glad 
and Ljung [32]. A comprehensive treatment of the theory up through 1997 is given by Seron et al. [56]. 
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Bode’s original book [7] is still instructive, and a recent monograph describes the context in which the 
limitations described by the gain-phase relation arose [49]. An excellent perspective on the importance 
of the theory of fundamental design limitations was given by Gunter Stein in the inaugural Bode Lecture 
in 1989, reprinted in [62]. 
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11.1 Introduction 


At first glance, the notion of deterministic uncertainty may seem to be a contradiction in terms—after all, 
the word “deterministic” is often used to signify the absence of any form of uncertainty. We will explain 
later on that, properly interpreted, this choice of words does indeed make sense. For the moment, we 
concentrate on the notion of uncertainty. 

Uncertainty in the control context comes in two basic versions—uncertain signals and uncertainty in 
the way the plant maps input signals into output signals (“plant uncertainty”). 

Most processes we wish to control are subject to influences from their environment—some of them 
known (measured disturbances, reference inputs), others uncertain signals (unmeasured disturbances, 
and noise corrupting measurements). All these signals are labeled “external,” because they originate in 
the “outside world.”* 

A plant model, by definition, is a simplified representation ofa real system, and is usually geared toward 
a specific purpose. Models that are meant to be used for feedback controller design tend to be especially 
crude. This is because (1) most popular design techniques can handle only very restricted classes of models 
and (2) in a feedback configuration, one can potentially get away with more inaccurate models than in 
applications that are based on a pure feedforward structure. 

Meaningful analysis and design, however, are possible only if signal and plant uncertainty is, in some 
sense, “limited.” In other words, we have to assume that there exists some, however incomplete, knowledge 
about signal and plant uncertainty—we need an uncertainty model. Such an uncertainty model defines an 
admissible set of plant models, G, and an admissible set of uncertain external input signals, W (Figure 11.1). 
The adjective in “deterministic uncertainty model” points to the fact that we do not attempt to assign 
probabilities (or probability densities) to the elements of the sets G and VV—every element is considered 
to be as likely as any other one. Based on this, one can ask the key (robustness) questions in control 


* Control inputs, on the other hand, are generated within the control loop, and are called “internal input signals.” 


11-1 


11-2 Control System Advanced Methods 


*—————— Known external inputs w; 


+m — GEG + Uncertain external inputs w, « W 


Output signals 
+*—— Control inputs u 


FIGURE 11.1 Signal and plant uncertainty models. 


systems analysis: Is closed-loop stability guaranteed for every plant model in G? Do desired closed-loop 
performance properties hold for every external input in W and every G € G? 

One has to keep in mind, however, that no class of mathematical models is able to describe every detail 
of reality, that is, the physical plant is not a mathematical model and therefore cannot be an element in G. 
Robustness of a desired closed-loop property with respect to any specific plant uncertainty model does 
not therefore guarantee that the real control system will also have this property. If the uncertainty model 
is chosen in a sensible way (that’s what this chapter is about), it will, however, increase the likelihood for 
the real system to “function properly.” 

We will first discuss signal uncertainty. Then, we will summarize the most common plant uncertainty 
models. Finally, we will briefly mention the topic of model validation, that is, whether a given set of 
experimental data is compatible with given uncertainty models G and W. We will work in a continuous- 
time framework, and all signals will be real and (except for some of the examples) vector valued. 


11.2 Characterization of Uncertain Signals 


Formulating a signal uncertainty model almost always involves a trade-off between conflicting principles: 
One wants the model to be “tight,” that is, contain only signals that make physical sense; tightening 
uncertainty, however, typically implies imposing additional restrictions on the model—it gets unwieldy 
and more difficult to use for analysis and design purposes. 
The following two widely used uncertainty models are on the extreme (simple) end of the spectrum. 
The first is 
W2(c) = (walt) | Iwull2<c}, (11.1) 


0° 1/2 
Iwate = (f wu(f u(t dt) 


that is, the admissible input set consists of all signals with £2-norm (energy) less than or equal to a given 
constant c; the second is 


where the norm is 


Woolc) = {wult) | lwulloo <¢}, (11.2) 
where the norm is 


|Wulloo = sup max |w,,(£)], 
t 1 


that is, the admissible input set consists of all signals with £..-norm (maximum magnitude) less than or 
equal to a given constant c. If necessary, these models can be refined by introducing suitable weights or 
filters: In this case, admissible input signals are 


W,(t) = 1s W(t — t)w,(t) dt, (11.3) 


where w,(f) ranges over the sets (Equation 11.1 or 11.2). Even such a modified uncertainty description 
remains pretty crude. Whether it is adequate depends on the control problem at hand. If the answer turns 
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out to be “no,” one has to cut back the “size” of the admissible signal classes by bringing in additional a 
priori information. This is illustrated in the following example. 


Example 11.1: 


Suppose we want to control room temperature. Clearly, outdoor temperature, To, is a disturbance 
signal for our control problem. Assume that, for one reason or another, we cannot measure To. In 
Toronto, To can go up to +30°C in summer and down to —30°C in winter. In this case, a simple 
uncertainty model is given by 


To(t) € Woo (30°) = {wult) | |lWulloo < 30°C}. 


Clearly, this set contains many signals that do not make physical sense; it admits, for example, 
temperatures of +30° during a winter night and —30° at noon in summer. A tighter uncertainty 
set is obtained if we write To(t) = Tg(t)+ Ta(t), where Tg(t) represents the average seasonal and 
daily variation of temperature, and T,(t) is a deviation term. The signal Tg(t) could, for example, 
be modeled as the output of an autonomous dynamic system (a so-called exosystem) with a pair of 
poles at s = +/(365 days)—' (accounting for seasonal variations) and s = +j(24 h)~1 (accounting for 
daily variations). For the deviation term, we can now assume a much stricter £o9-norm bound, for 
example, 10°. Furthermore, it makes sense to restrict the maximal rate of change of T, (t) to, say, 
5°/h. This is achieved if we define the admissible set of deviations via 


t 
Ta(t)= f W(t — t)wy(t) dt, 
—0o0 


where the weighting function W(t) is given by 


Se t/2, t>0 


w= | 0 t<o0 


(with t in hours) and w,(t) lives in the set Woo(1°). 


11.3 Characterization of Uncertain Plant Models 


Sometimes the plant can be modeled by a finite set, say by linearizing at a finite number of operating 
points. Then a controller can be designed for each plant in the set. Implementation involves a table lookup 
for which controller to apply, together with a way of smoothly transitioning when a switch is required. 

More commonly in robust control design, one tries to characterize the plant family G by specifying a 
nominal plant model together with a family of perturbations, denoted D, away from the nominal. As for 
the set of admissible input signals, we have two requirements for D: 


1. D should be tight (i-e., contain only perturbations that somehow “mirror” the difference between 
our nominal model and the real system). If we include perturbations that make no physical sense, 
we will end up with a controller that tries to accommodate “too many” models and might therefore 
be “too conservative.” 

2. D should be easy to use for analysis and design purposes. 


Again, both requirements rarely go together. By imposing structure on a perturbation model, we will in 
general be able to capture uncertainty more precisely; unstructured perturbation models, on the other 
hand, are in general easier to use in design procedures. 

For the lack of space, we can only present the most common plant uncertainty models—the ones that are 
widely used for controller design and synthesis purposes. It is not surprising that, in a sense, this purpose 
reflects back onto the model itself: Some models we will be dealing with involve restrictive assumptions 
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that can only be justified through a utilitarian argument—without these assumptions it would be a lot 
harder (or even impossible) to give robustness tests. We will look at unstructured perturbation models 
first. Such models are typically specified by giving, at every frequency, an upper bound on the maximum 
singular value of the transfer function matrix, this being a matrix generalization of the magnitude of a 
complex number. Then, we will deal with a specific class of structured perturbations that can be graphically 
characterized by Nyquist arrays. Finally, we will discuss a fairly general class of structured perturbations 
that includes the others. 


11.3.1 Unstructured Plant Uncertainty Models 


In what follows, G(s) will denote the nominal plant model, G,(s) the transfer function matrix ofa perturbed 
plant model, and A(s) the transfer function matrix of a perturbation. The dimensions of G(s) are p x q. 
It is assumed that the elements of G(s), G(s), and A(s) are proper real-rational transfer functions. The 
number of unstable poles of G and G,, that is, those in the closed right half-plane, are denoted mg and mg,. 


11.3.1.1 Additive Modeling Perturbations 
Additive perturbations are defined by (see Figure 11.2) 


G;(s) := G(s) + Aag(s). (11.4) 
The class of unstructured additive perturbations we will be looking at is given by 
Da i= {Aa | o[Ag(jo)] < 14(o); MG, =mg}. (11.5) 


Thus, the set of admissible A, is characterized by two assumptions: (1) for each frequency w, we know a 
(finite) upper bound [4 (o) for the size of A, (in the sense of the maximal singular value); (2) A4 cannot 
change the number of unstable poles of the model. Thus J4 is an envelope function on the size of the 
perturbation. The regularity of the function /,4 is not too important; it can be assumed to be, for example, 
piecewise continuous. The second assumption seems pretty artificial, but is needed if we want to give 
simple robustness tests.* It trivially holds for all stable perturbation matrices. Obviously, 4 (w) being finite 
for every w implies that Ay does not have any poles on the imaginary axis. A typical perturbation bound 
14() is shown in Figure 11.3: The DC gain is usually known precisely (A,4(0) = 0); at high frequencies, 
we often have G, + 0, G— 0 (G, and G are strictly proper), and therefore Ay — 0. 

The following example illustrates how an additive perturbation model typically can be obtained from 
frequency-response data. 


A,(s) 


FIGURE 11.2 Additive model perturbation. 


* A stability robustness test was first reported in Cruz et al. (1981). 
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FIGURE 11.3 Typical bound for additive modeling perturbation. 


Example 11.2: 


For simplicity, assume the plant is single-input/single-output (SISO). Suppose that the plant is stable 
and its transfer function is arrived at by means of frequency-response experiments: Magnitude and 
phase are measured at a number of frequencies, w;,i=1,...,M, and this experiment is repeated 
several, say N, times. Let the magnitude-phase measurement for frequency «; and experiment k 
be denoted (Giz, bj). Based on these data, select nominal magnitude-phase pairs (Gj, ;) for each 
frequency oj, and fit a nominal transfer function G(s) to these data. Then fit a weighting function 
I4(s) so that 


Gel Pk — Gel®} < |Iq(w;)|,  i=1,...,M; k=1,...,N. 


The next example shows how to “cover” a parameter-uncertainty model by an additive perturbation 
model. 


Example 11.3: 


Consider the plant model 


k 
—, 5<k<10. 
s+1 


Thus, the gain k is uncertain. Let us take the midpoint for the nominal plant: 


75 


G(s) = a 


Then the envelope function is determined via 


—G(j Iy(w), 5<k<10, 
i (io) <a) <k< 


that is, /4(@) = |2.5/(jw + 1)|. 


11.3.1.2 Multiplicative Perturbations 


Multiplicative (or proportional) perturbations at the plant output are defined by the following relation 
between G and G, (see Figure 11.4): 


G(s) := Up + Am(s)) G(s). (11.6) 


Such proportional perturbations are invariant with respect to multiplication (from the right) by a known 
transfer function matrix (e.g., a compensator). 
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G,(s) 


y 
> 
2 


FIGURE 11.4 Multiplicative (proportional) model perturbation. 


We consider the following class of (proportional perturbation) models: 


Du :=Dmi U Dy, (11.7) 
Dm := {Am | s[Am(jo)] <Iu(w); Aw stable } , (11.8) 
Du := {Am | 6[Am(jo)] < Iu(); me, = meg I: (11.9) 


Hence, admissible perturbations are either stable or do not change the number of unstable poles of 
the plant model. Again, in both cases, we assume that we know an upper bound for the perturbation 
frequency response Ajj(jw). A typical perturbation bound Jy is shown in Figure 11.5: Exact knowledge 
of DC gain implies Ay4(0) = 0; neglecting high-order dynamics often causes Ay to be greater than 1 at 
high frequency. 

The perturbation class Dg is of a simpler form than Equations 11.7 through 11.9, because an additive 
stable perturbation transfer function matrix does not affect the number of unstable poles. As the following 
example shows, this is not always true for multiplicative perturbations: A stable proportional perturbation 
can cancel unstable model poles. For example, 


G(s) = —-, Ay(s)= —, 
2 eS re 
G,(s) = [1+ Am(s)]G(s) = —. 
()=[1+ AmG)IG) = - re 
Stability robustness results for this class of pertubations have been given in Doyle and Stein (1981). 
11.3.1.3 Coprime Factor Perturbations 


For this uncertainty description, we need the concept of a left coprime factorization over RHoo* 
(Vidyasagar, 1985). Here, we will give only a very brief introduction to this subject. Details can be 
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FIGURE 11.5 Typical bound for multiplicative model perturbation. 


* RHoo denotes the set of proper stable real-rational transfer functions. 
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found elsewhere in this book. Consider a finite-dimensional linear time-invariant (FDLTI) SISO system. 
Clearly, its transfer function, G(s), can be represented by a pair of polynomials with real coefficients— 
Np(s), a numerator, and Mp(s), a denominator polynomial: G(s) = Np(s)/Mp(s). Furthermore, Np(s) 
and Mp(s) can always be chosen to be coprime, meaning that all common divisors of Np(s), Mp(s) are 
invertible in the set of polynomials (i-e., are real constants). Thus, there is no pole-zero cancellation when 
forming Np(s)/Mp(s). 

For several reasons, it proves to be an advantage to use a straightforward generalization, and to 
replace polynomials by proper stable transfer functions (ie., elements from the set RH»): We write 
G(s) = N(s)/M(s), where N(s), M(s) € RHoo can always be chosen to be coprime. This is called a coprime 
factorization over RHoo. Coprimeness in the RH.-context means that all common divisors are invertible 
in RH oo (i-e., stable, minimum-phase, biproper transfer functions). Hence, when forming N(s)/M(s), no 
cancellation of poles and zeros in the closed right half-plane can occur. Obviously, such a factorization is 
nonunique. However, it can be made unique up to sign by requiring |N(ja)|? + |M(jw)|? = 1, that is, the 
1 x 2 matrix [N (s) M (s)] is allpass. This is called a normalized coprime factorization (Vidyasagar, 1988). 

These concepts can be easily extended to the multivariable case. However, as matrix multiplication is 
not commutative, we have to distinguish between left and right coprime factorizations: A pair N(s), M(s) 
of stable transfer function matrices with appropriate dimensions is called a left coprime factorization of 
G(s) if G(s) = Mo} (s)N (s), and all common left divisors of N (s) and M (s) are invertible in RH. Similar 
for the concept of right coprime factorization. 

Now we are in a position to define coprime factor perturbations. These are additive stable pertur- 
bation terms in both the numerator and the denominator of a left (or right) coprime factorization (see 
Figure 11.6): 


G=M"'N; M,N...left coprime 


G; = (M+ Am)! (N+ An); M,, N,; ... left coprime. (11.10) 
—_—_e_ FFG 
M;' N, 


Specifically, we will consider the following class of unstructured coprime factor perturbations: 


Dun :={[Am Ay] |4[Am(jo) An(jo)] <Iun(@); Am, Ay stable} . (11.11) 


A typical perturbation bound Ijyn(w) may look as shown in Figure 11.3. Restricting the class Dyyy to stable 
transfer function matrices does not imply any loss of generality, as both M,N (denominator, numerator of 
the nominal model) and M+A M> N+A wn (denominator, numerator of any perturbed model) are stable 
by definition. Note that we do not have any restrictive assumptions regarding the number of unstable 
poles of admissible models G,(s): Perturbations from the set Dyyy can both increase or decrease the 


FIGURE 11.6 Coprime factor perturbation. 
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number of unstable model poles. For example, 


1 oe tT 
Oe | a 
ste s+l1 s+l 


—_——Sj{X>*_—— 
M(s)7! N(s) 
Meso Aye 
=U, a e— a 
2 ue stl 
1 s—-e] 1 
G,(s) = = : 
r{s) SE Ea Ea 
—_—>S_—=>_ Sr 


My(s)-!—N,(s) 


Coprime factor perturbations are especially useful for describing uncertainty in flexible structures. 
They allow covering a family of transfer function matrices with slightly damped uncertain pole-pairs by 
a (relatively speaking) small perturbation set. 


Example 11.4: 


Consider a nominal transfer function 


10 


Gls) = F005 — 5s +005 +5)" 


Suppose a perturbed model has a slightly different pair of poles: 


10 


Gr(s) = ; 
(8) = 50.05 — 6s 4.0.05 + 6) 


We first determine the magnitude of the smallest additive perturbation that, centered around G(s), 
covers Gy(s), and scale it (at each frequency) by 1/|G(jo)| (this is of course equivalent to computing 
the magnitude of the multiplicative perturbation connecting G(s) and G;(s)). The result is shown in 
the left part of Figure 11.7. 


We now compute normalized coprime factorizations, {N(s), M(s)} and {N,(s), Mr(s)}, for G(s) and 
Gr(s), and plot the size (maximal singular value) of the perturbation matrix 


[Nr (jo) —NU@) M,-(jo)— M(jo)] 
over frequency. This gives a relative (scaled) measure of perturbation size, as both [N(s) M(s)] and 
[Nr(s) Mr(s)] are allpass. It is shown in the right half of Figure 11.7. 


Clearly, there is an order of magnitude difference between the peak values of both perturbations. 


40 
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FIGURE 11.7 Perturbation sizes (normalized coprime factorization case shown on the right). 
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FIGURE 11.8 Generalized unstructured perturbation. 


(Normalized) coprime factor perturbations and their use in Hjo-design methods have been extensively 
investigated in McFarlane and Glover (1989). 


11.3.1.4 Generalized Unstructured Perturbations 


Additive, multiplicative, and coprime factor uncertainty can be conveniently represented in a unified 
way—via generalized unstructured perturbations (McFarlane and Glover, 1989). For this purpose, we 
introduce a “new” (proper) transfer function matrix 


(11.12) 


eG Be ean 


O21(s) ©r2(s) 


Its partitioning implies a partitioning of its input and output vector. The lower parts of both vectors 
represent the “usual” plant input and output signals. The upper part of the output signal, za, is fed back 
into the upper part of the input signal, wa, via a perturbation A(s) (Figure 11.8). Denote the number of 
rows and columns of A by ma and [,, respectively. Then, the transfer function matrix G, of the perturbed 
model is given as an Upper Linear Fractional Transformation of A with respect to ©: 


G, = @x + On A(T, — OA) O12 (11.13) 
= On + O21 (Im, — A@11) | A@x2. (11.14) 
For A = 0 (zero perturbation), we expect to recover the nominal plant model as the transfer function 


matrix between the plant input and the output vector. Hence, we must have ©27 = G. For Equations 11.13 
and 11.14 to make sense, we also have to assume that 


1. (Ii, — ©11(00) A(0o)) (or, equivalently, (lin, — A(0o)@11(00))) is nonsingular (i.e., G, is uniquely 
defined and proper) for all admissible perturbations. 

2. The transfer function matrix ©(s) is stabilizable from the plant input and detectable from the plant 
output. 


Specifically, we consider the following class of perturbations: 


Di= D U Dp, (11.15) 
D, := {A |6[A(jw)] < l(w), A stable}, (11.16) 
Dz := {A | 6[A(jw)] < lw), me, = mc} (11.17) 


We admit all bounded (on s = jw) perturbation transfer function matrices that are either stable or do not 
change the number of unstable poles of the plant model. 

We have not specified yet how to choose © if we want to translate additive, multiplicative, and coprime 
factor uncertainty into this general framework. By substituting into Equation 11.13, it is easy to check that 
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¢ for additive perturbations, 


0 iL 
e= Fy = 11.18 
} lp “| A (11.18) 
¢ for multiplicative perturbations, 
0 G 
O= », A=Aym, 11.1 
i; c| —_ 


=Mo*] [=G 
O= 0 ly dts =A” Axl: (11.20) 


For additive uncertainty, D} C D2 (stable perturbations do not change the number of unstable model 
poles). Hence, in this case, D and Dy are the same. In the case of coprime factor uncertainty, all pertur- 
bation terms are stable by definition. We can therefore write D = Dyw without restricting generality. 

This framework is especially useful for theoretical investigations. Instead of proving, say, robust stability 
for the three sets of perturbations D4, Dy, and Dyn, it suffices to establish this result for the class D of 
generalized perturbations. 


11.3.1.5 Which Unstructured Perturbation Model? 


All of the above perturbation models are reasonably simple and easy to use in design procedures. Hence, 
the decision on which model to choose hinges primarily on their potential to cover a given set G without 
including too many “physically impossible” plant models. In other words: we want the least conservative 
perturbation set that “does the job.” Whether this is a set of additive, multiplicative, or coprime factor 
perturbations depends on the problem at hand. A useful rule of thumb is: Whenever one deals with 
low-damped mechanical systems, it is a good idea to try coprime factor uncertainty first (compare the 
example in Section 11.3.1.3). For stable nonoscillatory processes, one might try an additive perturbation 
model first—it is more intuitive, and, if the nominal model is stable, excludes any unstable G,(s). 


11.3.2 Structured Plant Uncertainty 
11.3.2.1 Uncertain Parameters 


Physical (or theoretical), as opposed to experimental, model building usually results in a state model, 
and model uncertainty is often in the form of parameter uncertainty: The (real-valued) parameters of the 
state model have physical meaning and are assumed to lie within given intervals. When converting to a 
transfer function matrix, however, one usually gets a fairly complicated set of admissible parameters. This 
is due to the fact that, in general, each parameter of the resulting transfer function matrix depends on 
several parameters of the underlying state model. Only in exceptionally simple cases (as in the following 
example) can we expect the set of admissible parameters to form a parallelepiped. 


Example 11.5: 


Consider the spring—damper system in Figure 11.9. Force is denoted by u, position by y. Newton’s 
law gives the transfer function 


1 
s) = ———_—__ 
) ms2+ds+c 
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FIGURE 11.9 Spring-damper system. 


In most cases, one cannot expect to know the precise values of mass, damping, and spring 
constant. Hence, it makes sense to define a class of models by stating admissible parameter intervals: 
Am €[Muy —M, Mp — mM] := Dm 
Ad e[dy = d, do = d| = Dad 
Ac €lCy —¢, Co —C] := De 


1 
Ginde *= (m+ Am)s2 + (d + Ag)s +(¢+ Ac) 


Am Dm, Aqgé€ Pa Ace Det. 


11.3.2.2 Independent Additive Perturbations in the Elements 
of a Transfer Function Matrix 


Another straightforward (and often very useful) structured uncertainty description is the following: 
Consider an additive perturbation matrix A,(s) = G,(s) — G(s) and give frequency-dependent bounds 
],(@) for the magnitude of each of its elements: 


[Aan (j@)| ...  [Aa,, (7) Ki(m) ... lg) 

: yy : Sel 3 +. (11.21) 
[Aan Gol «+. [Adgg (Fo)! Ip(o) .-- Mpg) 
~~ =Ho) 


(<¢ denotes “elementwise less or equal”; |A,|¢ is a matrix with entries |A,,,|). Again, we make the 
additional assumption that no admissible perturbation shall change the number of unstable poles. Hence, 
we get the following classes of perturbations and plant models: 


Dae = {Aa| | Aa(jo)le <e L(w);_ me, = mc} (11.22) 
Gae = {G+Ay4|Ag € Dae}. (11.23) 


Such a set of models can be easily represented in a graphical way (Figure 11.10): Consider first the 
Nyquist plots of each element giz (jw) of the nominal model (“Nyquist array”). For each frequency, draw 
a circle with radius ]j,(@) around gix( jw). This gives a set of bands covering the nominal Nyquist plots. 
Then, a transfer function matrix G, isa member of the model set (Equation 11.23) ifand only ifthe Nyquist 
plot of each element g;,, (ja) is contained in the appropriate band, and G; and G have the same number 
of unstable poles. The usefulness of this uncertainty model for controller design purposes stems mainly 
from the fact that it fits “naturally” into the framework of Nyquist-array methods. Stability robustness 
results can be found in Owens and Chotai (1984) and Lunze (1984). 


11.3.2.3 Generalized Structured Plant Uncertainty 


A more general structured perturbation model has become very popular, as the so-called \1-theory (Doyle, 
1982, 1985) provides analysis and synthesis tools to deal with such uncertainty sets. As a motivation for 
this general perturbation model, consider the following example from Maciejowski (1989). 
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Imag 


FIGURE 11.10 Graphical representation of the model class Gye. 


Example 11.6: 


Let G(s) be a 2x 2 plant model with additive unstructured uncertainty A,(s) and independent 
multiplicative perturbations 84 (s), 82(s) in each input channel (Figure 11.11). Assume that A,(s), 54(s), 
and 89(s) are stable, proper, and bounded by 


[31 (Jo)| <1 (o), 
[32(Jo)| < I2(w), 
S[Aaljo)] < I3(o). 


It is easy to see that we could subsume all three uncertainty terms in a single (additive) perturbation 
matrix A,(s): 
% 0 
Gr=(6+Aa(b+| : 2) 
0 &) 


143, 0 310 
—G A Lica G ad . 
+( al 0 AG EF °]) 
———————— ee 


V1 


J2 


FIGURE 11.11 Example for a model with independent perturbations. 
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FIGURE 11.12 Generalized structured perturbation model. 


Using the properties of singular values, we can derive both structured and unstructured bounds for 
the overall perturbation Ay: 


eed diem Ser apy aeca WT BHI) 0 --, y=) 1a) 0 
aIBa(joll < 14/015 0 1 +Byjoy] tt6U019| 0 ‘9 


< I3(w) (1+ max(! (@), /2(@))) + [G(jo)] max(Iy (w), 12(w)) 
or 
[A aik(icr)| < 13(@)(1 + I ()) + Igix Go) lk ().- 


In both cases, the perturbation bounds will probably be very conservative, that is, the resulting 
uncertainty class will cover more than the perturbations A,(s), 3 (s), and 8o(s) we started off with. 
It is therefore a good idea to preserve perturbation structure when combining different “sources” 
of model uncertainty. This can be accomplished in the following way: As in Section 11.3.1.4, we 
define a “new” transfer function matrix ©(s) that—apart from the plant input and output—contains 
another pair of input and output vectors, wa and Z,.Z, is fed back into Ww, via a blockdiagonal 
perturbation matrix. Graphically, this corresponds to “pulling out” all perturbations from Figure 11.11 
and rearranging them in a blockdiagonal structure (Figure 11.12). 

Note that Figure 11.12 looks like Figure 11.8—the only difference being the blockdiagonal struc- 
ture of the perturbation matrix in the feedback loop—that mirrors the structure of the underlying 
perturbation model. 


In the general case, one proceeds in exactly the same way: The class of admissible models is represented 
by an Upper Linear Fractional Transformation of a blockdiagonal perturbation transfer function matrix 
A,(s) with respect to a suitably defined transfer function matrix ©(s): 


G; = Fu(O, As). (11.24) 


Structure and dimension of A, depend of course on the number of independent perturbation terms and 
their dimensions. Without loss of generality, we can assume that scalar uncertainty is always listed first 
in A,. In this general framework, we can also restrict A, to be stable: If unstable perturbations have to be 
considered, we can always circumvent this problem by introducing a coprime factorization with stable 
perturbations. Often, notation is simplified by normalizing the size of the perturbation blocks. This can 
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be easily done, if each perturbation bound ];(w) can be written as magnitude of a frequency response 
w;( joo): In this case, we just have to multiply ©); and © 2 from the left (or ©; and ©2) from the right) 
by a suitably dimensioned diagonal matrix containing the w;(s). As only magnitude is important, we can 
always choose the transfer functions w;(s) to be stable and minimum-phase. Hence, we get the following 
class of perturbations: 


dit, 0 ake 0 
0 
Hse Na | Aes deci, , A, stable, 6[A,(jo)] <1}. (11.25) 
Ak+1 
0 
0 0 A, 


Example 11.7: 


Let us look at the previous example again: Suppose w}(s), w2(s), and w3(s) are stable minimum-phase 
transfer functions with 


[wy (jo)| = h(a), 
|w2(jo)| = /2(o), 


|w3(jo)| = /3(@). 


Then we get 
| 0 o| mews : v2) 
Q(s) = w3lz 0 w3!2 
[G 1] G 
s. 08 0 8 
Seg Gall) OF 28a 20 
0 O A 
0 0 : 
8 0 0 
= 0) w2 
O00; = 
0 0 wae 


and—as expected — 


7 143; . 0 31.90 
=6+(as| 0 ox |e’ °)): 


All structured perturbation models in this section can be written as in Equations 11.24 and 11.25. 
However, when “translating” parameter perturbations into this framework, one has to take into account 


that D, admits complex perturbations, whereas parameters (and therefore parameter perturbations) in a 
state or transfer function model are real. 
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11.4 Model Validation 


Model validation is understood to be the procedure of establishing whether a set of experimental data is 
compatible with given signal and plant uncertainty models, W and G. It is not the (futile) attempt to show 
that an uncertainty model can always explain the true plant’s input/output behavior—future experiments 
might well provide data that are inconsistent with W and G. 

Model validation is a rapidly expanding area, and—for the lack of space—we can only hope to give a 
flavor of the subject by looking at a comparatively simple version of the problem: We assume that 


1. Wisa singleton (i.e., there is no signal uncertainty). 

2. Plant uncertainty is in the form of stable generalized unstructured perturbation (i.e, G;(s) = 
Fu(@(s), A(s)), where A(s) € D,—see Section 11.3.1.4), and @21(s) is invertible. 

3. Experimental data are given in the form of M frequency-response measurements G;,(jw1),..., 
G,( jOm ). 


The plant uncertainty model of item 2 contains the cases of coprime factor perturbations and of stable 
additive and multiplicative perturbations. 
With the invertibility condition for ©) in force, we can rewrite the perturbed plant model as follows: 


G, = Fyu(@, A) (11.26) 

= O22 + O21 (Imy — AQ) AO (11.27) 
-1 

=(AT2+0x2) (ATi +P), (11.28) 


where 


Ee | = ig — 0105 O22 Bead (11.29) 


Ta Pol’ 07, Ox 0; 
This can be easily checked by substituting Equation 11.29 into Equation 11.28. Multiplying Equation 11.28 
by (AT 12 + I'22) from the left gives 


A (T12G, — Ty) = P21 — P'22G, . (11.30) 
—— —— 
=W :=U 
Consistency of experimental data G,(jw1),...,G,(jom) and uncertainty model is equivalent to the 
existence of a transfer function matrix A(s) € D, that solves Equation 11.30 for w ,..., wy. Clearly, a 


necessary condition for this is that the system of linear equations over C 
A; W(ja;) = U(ja;) (11.31) 


has a solution Aj; with o[ Aj] < I(w;) for eachi € {1,..., M}. Using interpolation theory, it has been shown 
in Boulet and Francis (1994) that this condition is also sufficient*. If suitable A; exist, one can always find 
a stable transfer function matrix A(s) such that A(jw;) = Ai, i=1,...,M, and o[A(jw)] < I(w) for all 
oe R. 


11.5 Further Reading 


Any book on robust control contains information on plant model uncertainty. We especially recommend 
Vidyasagar (1985), McFarlane and Glover (1989), Lunze (1989), Glad and Ljung (2000), and Skogestad 
and Postlethwaite (2005). Parts of this chapter are based on Raisch (1994). See also Doyle et al. (1992). 


* Their proof is for coprime factor perturbations. It carries over to the slightly more general case considered here. 
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On the problem of fitting a model to frequency-response data, see Hindi et al. (2002) and Balas et al. 
(2009), and from time-domain data see Volker and Engell (2005). 

Several papers related to model validation can be found in a special issue of the IEEE Transactions on 
Automatic Control dealing with “System identification for robust control design” (IEEE, 1992). 
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12.1 Introduction 


A linear system with white noise added to the input and, often but not always, white noise added to the 
output is the most common model for randomness in control systems. It is the basis for Kalman filtering 
and the linear quadratic Gaussian (LQG) or H2 optimal regulator. This chapter presents the basic facts 
about linear systems and white noise and the intuition and ideas underlying them. Results are emphasized, 
not mathematically rigorous proofs. It is assumed that the reader is familiar with the elementary aspects 
of probability at, for example, the level of Leon-Garcia [1]. 

There are many reasons why people so commonly use a linear system driven by white noise as a model 
of randomness despite the fact that no system is truly linear and no noise is truly white. One of the 
reasons is that such a model is both elementary and tractable. The calculations are easy. The results can 
be understood without a deep knowledge of the mathematics of stochastic processes. 

A second reason is that a large class of stochastic processes can be represented as the output of a linear 
system driven by white noise. The precise result can be found later in this chapter. The generality and 
tractability of this model can be quite dangerous because they often lead people to use it inappropriately. 
Some of the limitations of the model will also be discussed. 

Scalar discrete-time stochastic processes are described first, because this is the simplest case. This is 
followed by a discussion of the ways single-input single-output (SISO) discrete-time linear systems operate 
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on scalar discrete-time stochastic processes. Vector discrete-time stochastic processes and multiple-input 
multiple-output (MIMO) linear systems, a notationally more difficult but conceptually identical situation, 
are then briefly covered. 

The second half of this chapter describes continuous-time stochastic processes and linear systems in 
the same order as was used for discrete time. 


12.2 Discrete Time 


The only difference between an n-dimensional vector random variable and a scalar discrete-time stochastic 
process over n time steps is in how they are interpreted. Mathematically, they are identical. As an aid to 
understanding discrete-time stochastic processes, this equivalence will be emphasized in the following. 


12.2.1 Basics 


The usual precise mathematical definition of a random variable [2,3,9] is unnecessary here. It is sufficient 
to define an n-dimensional random variable by means of its probability density function. 


Definition 12.1: 


An n-dimensional random variable, denoted x = [X, X2...Xn]', takes values x € R" according to a proba- 
bility that can be determined from the probability density function (pdf) px(x). 


Unfortunately, the notation needed to describe stochastic processes precisely is very complicated. 
Thus, it is important to recognize that bold letters always denote random variables while the values 
that may be taken by a random variable are denoted by standard letters. For example, x is a random 
variable that may take values x. Underlined lower case letters will always denote vectors or discrete-time 
stochastic processes. Underlined capital letters will denote matrices. A list of all the notation used in this 
chapter appears at the end of the chapter. The reader is assumed to know the basic properties of pdfs. 


Definition 12.2: 


A scalar discrete-time stochastic process over n time steps, x(1), x(2),...,x(n), denoted x, takes values 
x(k) € R, where R denotes the real numbers and k =1,2,...,n, according to a probability that can be 
determined from the pdf px(x). 


The equivalence of the two definitions is obvious once x(1),x(2),...,x(m) is written as a vector 
[x(1) x(2)...x(n)]/ = [x x2...Xn]', where ’ denotes the transpose. The equivalence is emphasized by 
using the same notation for both. The context will make it clear which is meant whenever it matters. 

It is important to be able to specify the relationships among a collection of random variables. A simple 
special case is that they are completely unrelated. 


Definition 12.3: 


The n-vector random variable (equivalently, discrete-time stochastic process) x, is composed of n independent 
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random variables x, X2,...,Xn if and only if their joint pdf has the form 


px(x) = | | Px, (xe) = Px, (1) Px (2) - - Px (Xn): (12.1) 
k=1 


Independence is an extreme case. More typically, and more interestingly, the individual elements of a 
vector random variable or a scalar discrete-time stochastic process are related. One way to characterize 
these relationships is by means of the conditional pdf, px, jx, (*; |X). When px, (x2) 4 0, the conditional 
pdf is given by 

Px(x) 
Px, (x9) : 


Px, ix, (411%) = (12.2) 


where x, is the m— vector [x, x2...Xm]';X, isthe (n—m) —vector [Xm+1 Xm+2---Xn]';x is the n— 


vector [x] X2 .. .X,]’; and the x;s have the same dimensions as the corresponding x;s. It is often possible to 
avoid working directly with pdfs, especially in the study of linear systems. Instead, one uses expectations. 


12.2.2 Expectation 


Definition 12.4: 


The expected value, expectation, or mean, of a scalar random variable, x, is denoted by E(x) or m and 
given by 


mS E(x) & / S xpx(x) dx. (12.3) 


Applying Definition 12.4 to the scalar random variable x(k), the kth element of the scalar discrete-time 
stochastic process, x, gives 


m(k) © E(x(k)) = i uae / - x(k)px(x)dxy dx... dx. (12.4) 


Note that the integrations over xe, ¢ # k, simply give the necessary marginal density px 4)(x(k)). 
The computation in Equation 12.4 can be repeated for all k = 1, 2,..., and the result is organized as 
an n-vector 
E(x) = [E(x(1)) E(x(2))... E(x(n))I’. (12.5) 


One can also take the expectation with respect to functions of x. Two of these are particularly important. 


Definition 12.5: 
The covariance of the scalar random variables, x, and xg, is denoted by r;.¢, and given by 
A [o,@) CO 
rae ECC — mC — m(O)) = ff mG —MODPrclaer erase. (12.6) 
—-W J—-C 


Definition 12.5 can be applied to each pair x(k)x(¢),k, = 1,2,...,n of elements of the discrete-time 
scalar stochastic process x. The result is a collection of n* elements rye, k, = 1,2... n. It is conventional 
to emphasize the time dependence by defining 


r(k,£) © ree. (12.7) 
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It should be obvious from Equation 12.6 that 
r(k,2)=r(6k) forall k,2@ (12.8) 
and that (because px(x) > 0 for all x), 
r(k,k)>0 forallk. (12.9) 


In the context of discrete-time stochastic processes, r(k, €) is known as the autocovariance function of 
the stochastic process x. It can be helpful, in trying to understand the properties of the autocovariance 
function, to write it as a matrix. 


r(1,1) r(1,2) ... r(i,n) 
r(2,1)  _r(2,2) 

R= : : . ‘ (12.10) 
ree 1) wit me n) 


In the context of n-vector random variables, the matrix R is known as the covariance matrix. The 
autocovariance will be discussed further subsequently. Another important expectation will be defined 
first. 


Definition 12.6: 


The characteristic function of a scalar random variable, x, is denoted by fx(w) and given by 


fx(w) = E(e#*) = / ~ ep. (x) dx, (12.11) 


where 


jav-l. 


Note that f,(w) is a deterministic function of w, not a random variable. Note also that f,(—w) is the 
Fourier transform of px (x) and is thus equivalent to px(x) in the same way that Fourier transform pairs 
are usually equivalent; there is a unique correspondence between a function and its transform. 

The generalization to the case of n-vector random variables or discrete-time stochastic processes over 
n time steps is as follows. 


Definition 12.7: 


The characteristic function of an n-vector random variable (or discrete-time stochastic process), x, is denoted 
fx(@) and given by 


3 ao s 
flo) =Ele S| = Be), (12.12) 


The characteristic function is particularly useful for studying the effect of linear mappings on a stochas- 
tic process. As an example, consider the operation of an m x n real matrix L on the n-vector random 
variable x. 

Let y = Lx; 


flo) = E(e!*Y) = B(e!@') = E(e¥ 9) = f(L'a). (12.13) 
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12.2.3 Example: Discrete-Time Gaussian Stochastic Processes 


Definition 12.8: 


Ann-vector random variable (or discrete-time stochastic process), x, isa Gaussian (normal) random variable 
(discrete-time stochastic process) if and only if it has the n-dimensional Gaussian (normal) pdf 


1 1 1p-1 
— 3 (x—m)'R~*(x—m) 
Px) aa (2n)"/2(detRy'/2° 2 > (12.14) 


where 
m = E(x), 


R=E((x—m)(x—m)), _ the covariance matrix of x. 


The definition implies that R is symmetric (R = R’, see Equations 12.8 and 12.10 and that R must be 
positive definite (y’Ry > 0 for all n-vectors y 4 0 and often indicated by R > 0). 

It is easy to demonstrate that the characteristic function of the n-vector Gaussian random variable, x, 
with mean m and covariance matrix R is 


fxl@) = elem 10'Ro (12.15) 


Any n-vector random variable or stochastic process that has a characteristic function in the form 
of Equation 12.15 is Gaussian. In fact, some authors [2] define a scalar Gaussian random variable by 
Equation 12.15 with m and R scalars, rather than by Equation 12.14. The reason is that Equation 12.15 
is well defined when R = 0, whereas Equation 12.14 blows up. A scalar Gaussian random variable x with 
variance R = 0 and mean m makes perfectly good sense. It is the deterministic equality x = m. 

One other special case of the n-vector Gaussian random variable is particularly important. When 
ree = 0 for all k F 2, R is a diagonal matrix. The pdf becomes 


l 1( ane } (xg—my)* ieee va ut | 
= "11 £29. Tan 
Px(x) = n/p 1/2.1/2 1/2 
(2m) ri 15) + Tan 


1 _ 1 Gy =m)? 1 ok (xg—mg)? 1 1 (a—mn)? (12.16) 
ae ee Sie es oak (ar ae 
V 2014 V 2755 JV 27 nin 


= px, (x1 )Px> (x2) +++ Px, (Xn). 

In other words, the x; are independent random variables. It is an important property of Gaussian 
random variables that they are independent if and only if their covariance matrix is diagonal, as has just 
been proven. 

Finally, the characteristic function will be used to prove that if y = Lx, where L is an m x n real matrix 


and x is an n-vector Gaussian random variable with mean m and covariance R, then y is an m-vector 
Gaussian random vector with mean Lm and covariance LRL’. The proof is as follows: 


y=Es 
fy) = fa(L'o) 
by Equation 12.15 


— eiL’o)'m— 3 (L'0) RLw) = gio’ Lm—5w/LRL'o = ej (Lm)—50/(LRL')o (12.17) 


Finally, the uniqueness of Fourier transforms (characteristic functions), and the fact that Equation 12.17 
is the characteristic function of an m-vector Gaussian random variable with mean Lm and covariance 
matrix LRL’, completes the proof. 
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12.2.4 Stationarity, Ergodicity, and White Noise 


Several important properties of a discrete-time stochastic process are, strictly speaking, properly defined 
only for processes for which k = --- —2, —1,0,1,2,... (ie, —0o < k < 00). 
Let 


t= L..ck 4 See oe Skee) (12.18) 


denote either the scalar discrete-time stochastic process on the interval —oo < k < oo or the equiva- 
lent infinite-dimensional vector random variable. It is difficult to visualize and write explicitly the pdf 
for an infinite-dimensional random variable, but that is not necessary. The pdfs for all possible finite- 
dimensional subsets of the elements of x completely characterize the pdf of x. Similarly, E(x) = m is 
computable term by term from the x; taken one at a time and E((x — m)(x — m)') = R is computable 
from all pairs x;, xj taken two at a time. 


Example 12.1: 


Let x be a Gaussian scalar discrete-time stochastic process with mean E(x;,) = mz = 0 forall k, —co < 
k < oo and covariance 


1 k= 
E(x,Xg) =r(k,€)= 41/2 |k-£)=1 
0 otherwise. 


Note that this completely describes the pdf of x even though x is infinite-dimensional. 


This apparatus makes it possible to define two forms of time-invariance for discrete-time stochastic 
processes. 


Definition 12.9: 


A scalar discrete-time stochastic process x, defined on —0o0 < k < 00 is stationary if and only if 


Py (xz Xkoo ++ +9 Xk ) 

ky 9 KES 9's Ke 12> %kz> »Xky 

ne? P (12.19) 
= Pru pep Sky +kp> +o Xkntkp (Xk +hp» Xko-tkp» see »Xky-tkp) 


for all possible choices —00 < ke, kp < 00, €=1,2,...,n and all finite n. 


The region of definition of stochastic processes defined on (—00, 0c) will be emphasized by using the 
notation X(_¢0,99) to denote such processes. 

It can be difficult to verify Definition 12.9. There is a weaker-and-easier-to-use form of stationarity. It 
requires a preliminary definition. 


Definition 12.10: 


A scalar discrete-time stochastic process, x, is a second-order process if, and only if, E(xz) < 00 for all 
k, 00 <k < 00. 
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Definition 12.11: 


A second-order scalar discrete-time stochastic process, X(_¢0,90) is wide-sense stationary if, and only if, its 
mean m and autocovariance r(k, £) satisfy 


m(k) =m, aconstant for allk, —oo<k<o, (12.20) 
rk Q)=r(k+i,€+i) forallk,£i, —oo<k,hi<o. (12.21) 

It is customary to define 
r(k) . rk+2,0) forallk,£, —oo<k,l<om. (12.22) 


It is obvious that a stationary discrete-time stochastic process is also wide-sense stationary because the 
invariance of the pdfs implies invariance of the expectations. Because the pdf of an n-vector Gaussian 
random variable is completely defined by its mean, m, and covariance, R, a scalar wide-sense Gaussian 
stationary discrete-time stochastic process is also stationary. 

The apparatus needed to define discrete-time white noise is now in place. 


Definition 12.12: 


A scalar second-order discrete-time stochastic process, X(—o0,90); is a white noise process if and only if 


mk) = E(xp)=0  forallk, —oo<k< oo, (12.23) 
r(k) = E(xe44xe) = r8(k) forall —cwo <k,l< oo, (12.24) 
where 
r>0 
and 
1 k=0 
&(k) = 12.25 
(k) f otherwise. ( ) 


If, in addition, x, is Gaussian for all k, the process is Gaussian white noise. An explanation of the term 
“white noise” requires an explanation of transforms of stochastic processes. This will be forthcoming 
shortly. First, the question of estimating the pdf will be introduced. 

In the real world of engineering, someone has to determine the pdf of a given random variable or 
stochastic process. The typical situation is that one has some observations of the process and some prior 
knowledge about the process. For example, the physics often indicates that a discrete-time stochastic 
process, x, —0o < k < o, is wide-sense stationary—at least as an adequate approximation to reality. If 
one is content with second-order properties of the process, this reduces the problem to determining m 
and r(k), —oo < k < 0, from observations of the process. The discussion here will be limited to this 
important, but greatly simplified, version of the general problem. 

In order to havea precisely specified problem with a well-defined solution, assume that the data available 
are one observation (sample) of the stochastic process over the complete time interval —oo < k < oo. This 
is certainly impossible for individuals having a finite lifespan, but the idealized mathematical result based 
on this assumption clarifies the practical situation. 
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Define 
: > (12.26) 
me = — Xk, : 
L oy) k 
k=—£ 
1 e 
r(k) = y 3; Xj 4EXi- (12.27) 
i=— 


Note that mg and r¢(k) are denoted by bold letters in Equations 12.26 and 12.27. This indicates that 
they are random variables, unlike the expectations m and r(k), which are deterministic quantities. It can 
then be proved that 


jim E[(m, — m)*] = 0, (12.28) 
00 
provided 
jim r(k) = 0. (12.29) 
—>0o 


A similar result holds for rg(k) (see [2, pp. 77-80]). 

In order to apply this result to a real problem one must know, a priori, that Equation 12.29 holds. 
Again, this is often known from the physics. 

The convergence result in Equation 12.28 is fairly weak. It would be preferable to prove that mg 
converges to m almost surely (with probability 1). It is possible to prove this stronger result if the process 
is ergodic as well as stationary. This subject is both complicated and technical. See [2,4] for engineering- 
oriented discussions and [3] for a more mathematical introduction. 


12.2.5 Transforms 


In principle, one can apply any transform that is useful in the analysis of discrete-time signals to discrete- 
time stochastic processes. The two obvious candidates are the Z-transform and the discrete Fourier 
transform [5]. There is a slight theoretical complication. To see this, consider the discrete Fourier trans- 
form of the discrete-time stochastic process X(_o0,00) 


x(Q) E> x(kye I, (12.30) 


k=—0o 


Note that xp(Q) is an infinite-dimensional random vector for each fixed value of Q (it is a function of the 
random variables x(k); so it is a random variable) and xp(Q) is defined for all Q, —oo < Q < 00, not just 
integer values of &2. In other words, x;() is a stochastic process in the continuous variable &. As is usual 
with the discrete Fourier transform, x¢({2) is periodic in Q with period 21. 

A very important use of transforms in the study of discrete-time stochastic processes is the spectral 
density function. 


Definition 12.13: 


Let X(_c0,00) be a wide-sense stationary scalar discrete-time stochastic process with mean m and autocovari- 
ance function r(k). Assume )-7-._ 45 |r(k)| < 00. Then the discrete Fourier transform of the autocovariance 
function r(k), 


—0O 


s(2) => r(keI*, (12.31) 


k=—00 


is well defined and known as the spectral density function of X(—o0,00) 
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u y 
> S [>_> 
Input Output 

System 


FIGURE 12.1 A symbolic representation of a discrete-time linear system, S, with input u=[...u—2 u_1 uo 1 ...]/ 
and output y =[...y—1 yo yi -- I. 


Note, as the notation emphasizes, that no part of Equation 12.31 is random. Both the spectral density 
and the autocovariance are deterministic descriptors of the stochastic process, X(_¢,00): 
The inverse of Equation 12.31 is the usual inverse of the discrete Fourier transform. That is, 


1 ; 
r(k) = — / s(Q)e**dQ, (12.32) 
2m Qn 
where /,, means the integral is taken over any interval of duration 2n. 
When m = 0, 

dQ 
ro) = Exp) =f tay. (12.33) 

2n 20 


IfX(_o0,90) is a voltage, current, or velocity, then r(0) can be interpreted as average power, at least to within 
a constant of proportionality. With this interpretation of r(0), s(Qo) (where {20 is any fixed frequency) 
must be the average power per unit frequency in X(_¢0,99) at frequency {29 [2]. This is why it is called the 
spectral density. 


Example 12.2: 


SUPPOSE X(_ 60,00) is a white-noise process (see Definition 12.12) with r(k) = 8(k). The spectral density 


function for this process is 
[o,@) 


(Q)= > ake IP =1. (12 34) 
k=—0o 
Hence it is called “white noise.” Like white light, all frequencies are equally present in a white noise 
process. A reasonable conjecture is that it is called “noise" because, in the early days of radio and 
telephone, such a stochastic process was heard as a recognizable and unwanted sound. 


12.2.6 Single-Input Single-Output Discrete-Time Linear Systems 


It is convenient, both pedagogically and notationally, to begin with SISO discrete-time linear systems 
described by their impulse response, h(k, 2). The notation is shown in Figure 12.1. 


Definition 12.14: 


The impulse response of a SISO discrete-time linear system is denoted by h(k, ¢), were h(k, €) is the value of 
the output at instant k, when the input is a unit impulse at instant £. 


The response of a linear system S to an arbitrary input, u(_oo,90), is then given by a convolution sum 


yk) = > h{k, Auld); -00 <k <0. (12.35) 


l=—0o 
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The crucial point is that S is a linear map, as is easily proved from Equation 12.35. Linear maps take 
Gaussian stochastic processes (random variables) into Gaussian stochastic processes. A special case of 
this, when S can be written as an n x m matrix, was proved earlier (Equation 12.17). 


Example 12.3: 


Consider a SISO discrete-time linear system, S, with input, (0,00), a discrete-time Gaussian stochas- 
tic process having mean m,(k), and autocovariance ry,(k, 2). Knowing that the system is linear and 
that the input is Gaussian, the output, y(_o0,00), Must be a Gaussian stochastic process; what are its 
mean and autocovariance? They can be calculated element by element: 


my (k) = E(y(k)) = E pa, h(k, )u a h(k, OE 
l=—00 
lee) 
SY hlk, Amul); —0o <k <0, (12.36) 
l=—00 
E((y(k) — mu(k))(y(Q) — mu(2))) 


(oe) 
So SS Atk, DAU JE ((wli) — mul) (UG) — mu()) 


j=—00 f=—oo 
(oe) (oe) 
= > YS Ak, Nhl, jruli,/). (12.37) 
i=—00 f=—00 


Of course, not every stochastic process is Gaussian. However, careful review of the previous example 
shows that Equations 12.36 and 12.37 are valid formulas for the mean and autocovariance of y(—o0,00) 
whenever U— 00,90) is a second-order scalar discrete-time stochastic process with mean m,(k) and autoco- 
variance r, (i,j) and the system S is asymptotically stable. This is a very important point. The second-order 
properties of a stochastic process are easily computed. The calculations are even simpler if the input pro- 
cess is wide-sense stationary and the linear system is time-invariant. 


Example 12.4: 


Consider a SISO discrete-time linear time-invariant (LTI) system, S, with input, U(—¢0,00), a Wide-sense 
stationary second-order discrete-time stochastic process having mean my and autocovariance ry(k). 


Denote the impulse response of S by h(k), where h(k) 2 h(k + ££) and assume that S is asymptotically 
stable. 


£=—0o (12.38) 
oe aw 
= ye A(QE(u(k — 2) = amu, 
b=-00 


where 
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Similarly, using Equation 12.37, and defining @=k-—iandk=2 -j 


5 x A h(k — h(E — fyruli —f) aS 3 A(Oh(k)ry(k — €— 0+ k). (12.39) 


i=—00 j=—00 b=—oo k=—00 


Note that the convolution sum in Equation 12 39 depends only on the difference k — ¢. Thus, Equa- 
tions 12 38 and 12.39 prove that my(k) = my, a constant, and that ry(k, ¢) = ry(k — £) = ry(k), where 


k 2k —£.This proves that y(_ 4,90) is also wide-sense stationary. 


Although the computations in Equation 12.39 still appear to be difficult, they clearly involve determin- 
istic convolution. It is well known that the Fourier transform can be used to simplify such calculations. The 
impulse responses of many discrete-time LTI systems, h(k), —oo < k < ov, have discrete Fourier trans- 
forms, hy (&2),0 < 2 < 21. The exceptions include unstable systems. Assume hy ({2) exists in the preceding 
example and that u(_oo,00) is wide-sense stationary with mean m, and spectral density function s(Q). 
From the definition of the discrete Fourier transform 


CO 
hp(Q)= Yo h(kye I, (12.40) 
k=—00 
it is evident that Equation 12.38 becomes 
My = hy (O)mu. (12.41) 


Using the inversion formula (Equation 12.32) in Equation 12.39 gives 


(oe) (oe) 


HOE >> monn (= i: s(apel@-t*.aq) 


l=—@ i=—0o 


ai Ss h(i)sy (Q) jeI2tk+i) (> 2S h(€)e mn) 3 dQ 


i=—0o L=—0o 
oe) 


i 2K dQ (12.42) 
={ ( Y> Ae! ) irons = 


i=—0o 
on AQ 
= | hy(—Q)hg(Q)sy(Q)e* — 
he y(—Dy-(D5y( De! 
on AQ 
= / [hp (Q)/?su(Q)eF* — 
2n 2m 
By the uniqueness of Fourier transforms, Equation 12.42 implies 
sy(Q) = |Ag(Q)|?su(Q). (12.43) 
SISO linear systems in state-space form generally have an n-vector state. When either the initial state 
is random or the input is a stochastic process, this state is a vector stochastic process. The ideas and 


notation for such processes are described in the following section. This is followed by a description of 
linear systems in state-space form. 


12.2.7 Vector Discrete-Time Stochastic Processes and LTI Systems 


The vector case involves much more complicated notation but no new concepts. 
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Definition 12.15: 


An m-vector discrete-time stochastic process over n time steps, denoted X . {[xi(k) xo(k)...Xm(K);3 k= 
1,2, ... n}, takes values X = {[x1(h) x(k)... xm(k))'sk = 1,2,...,n} according to a probability that can 
be determined from the pdf px(X). 


It can be helpful to visualize such a process as an m x n matrix 


xi(1) x(2) ... x(n) 
xX2(1) -x2(2) ... ) &X(n) 

X= . ; ; . (12.44) 
Xm(1) Xm(2) ... Xma(t) 


The notation is that bold capital underlined letters denote vector stochastic processes. 
All of the scalar results apply, with obvious modifications, to the vector case. For example, E(X) can be 
computed element by element from Definition 12.4. 


[oe 


E(xe(k)) = i eng tds (12.45) 


for all = 1,2,...,m,k = 1,2, ... n (see Equation 12.4). 
Then, the nm results of Equation 12.45 can be organized as an m-vector over n time steps, m(k) = 
[m1(k) m2(k) ...mm(k)]’, where 
me(k) = E(xe(k)). (12.46) 
Similarly, the autocovariance of X in Equation 12.44 can be computed element by element using 


Definition 12.5 and Equation 12.7. The results are conventionally organized as an autocovariance matrix 
at each pair of times. That is 


ri1(k, £) r12(k, £) ein rim(k, £) 
121 (k, £) ro2(I, £) hs ram k, £) 
ile £) SU iB) age ae t) 
where 
ri(k, €) 2 E((xi(k) — mi(k))(x;(€) — mj(2))). 
Example 12.5: 


Discrete-time m-vector white noise: Any white noise must be wide-sense stationary and have zero 
mean. Thus, letting &(k) denote an m-vector white noise at the instant k, 


ms (k) ) 


= E(&(k)) =0 forall k,-—oo<k<«o, 
Re (k, £) = E(&(k)§/(0)) =O forall k # £,-00 <k, £< &, 


(12.48) 


because white noise must have autocovariance equal to zero except when k = £. When k = £, the 
autocovariance can be any positive-semidefinite matrix. Thus, 


Re (k, £) = Re &(k — 2), (12.49) 


where Re can be any positive-semidefinite matrix (y’Rey > 0 for all m-vectors y). 
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Finally, as is customary with stationary processes, 
R.(k) = Re(K+60 = R8(k). (12.50) 


Reading vector versions of all the previous results would be awfully tedious. Thus, the vector versions 
will not be written out here. Instead, an example of the way linear systems operate on vector discrete-time 
stochastic processes will be presented. 

The previous discussion of linear systems dealt only with impulse responses and, under the added 
assumption of time invariance, Fourier transforms. Initial conditions were assumed to be zero. The 
following example includes nonzero initial conditions and a state-space description of the linear system. 
The system is assumed to be LTI. The time-varying case is not harder, but the notation is messy. 


Example 12.6: 


Let Xq be an n-dimensional second-order random variable with mean m,, and covariance R,,. Let 
& and @ be, respectively, m-dimensional and p-dimensional second-order discrete-time stochastic 
processes on 0 <k < ky. Let E(&) = 0,£(0) = 0, E(&(k)&’(0)) = BS(k — £), E(8(k)6’(2)) = ©8(k — 2), and 
E(&(k)6’(€)) = O for all 0 < k, £ < ky. Strictly speaking, E and © are not white-noise processes because 
they are defined only ona finite interval. However, in the context of state-space analysis it is standard 
to call them white noise processes. Finally, assume E((Xq — my, )&’(k)) = 0 and E((x9 — m,,)6’(k)) = 0 
for allO <k < kg. 

Suppose now that the n-vector random variable x(k + 1) and the p-vector random variable y(t) 
are defined recursively for k = 0,1,2,...,k¢ by 7 


x(k + 1) = Ax(k) + L8(k); x(0) = Xo, (12.51) 
y(k) = Cx(k) + 8(k), (12.52) 


where A is a deterministic n x n matrix, Lis a deterministic n x m matrix, and C is a deterministic p x n 
matrix. What can be said about the stochastic processes X = {x(k);_ k=0,1,2,...,k¢ +1} and Y= 
{y(k); k =0,1,2,...,kp}? 

~ The means and autocovariances of X and Y are easily computed. For example, 


E(x(k + 1)) = E(Ax(k) + L&(k)) = AE(x(k)) + LE(&(k)) = AE(x(k)). (12.53) 


Similarly, 
E(y(k)) = CE(x(k)). (12 54) 


Note that Equation 12 53 is a simple deterministic recursion for E(x(k + 1)) starting from E(x(0)) = 
Myo: Thus, 


E(x(k)) = m,(k) = AX m,,, (12.55) 


Ely(k)) = CA‘ m,,. (12.56) 


Similarly, recursive equations for the autocovariances of X and Y can be derived. Because the 
expressions for Y are just deterministic algebraic transformations of those for X, only those for X are 
given here. 

Ry(k + 1,k +1) = E((x(k + 1) — my(k + 1))(x(k + 1) — my (k + 1’) 
= E((A(x(k) — m, (k)) + L&(k))(A(«(k) — my (k)) + L&(k))’) 
= AE ((x(k) — m, (k))(x(k) — m(k))') A’ + LEL’ (12.57) 
= AR, (k, A’ + LEL’, 
Ry lk +1, k) © El(xtk + 1) — my(k + 1))(x(k) — my (K))’) = AR (k, k). (12.58) 
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Note that Equations 12.57 and 12.58 use the assumption that E((x(k) — my (k))& (4) = 0 for all 0 < 
k, £ < kr. Furthermore, the recursion in Equation 12.57 begins with R, (0,0) = R,,. 


12.3 Continuous Time 


Continuous-time stochastic processes are technically much more complicated than discrete-time stochas- 
tic processes. Fortunately, most of the complications can be avoided by the restriction to second-order 
processes and linear systems. This is what will be done in this article because linear systems and second- 
order processes are by far the most common analytical basis for the design of control systems involving 
randomness. 

A good understanding of the discrete-time case is helpful because the continuous-time results are often 
analogous to those in discrete time. 


12.3.1 Basics 


It is very difficult to give a definition of continuous-time stochastic process that is both elementary and 
mathematically correct. For discrete-time stochastic processes every question involving probability can 
be answered in terms of the finite-dimensional pdfs. For continuous-time stochastic processes the finite- 
dimensional pdfs are not sufficient to answer every question involving probability. See [2, pp. 59-62] for 
a readable discussion of the difficulty. 

In the interest of simplicity and because the questions of interest here are all answerable in terms of the 
finite-dimensional pdfs, the working definition of a continuous-time stochastic process will be as follows. 


Definition 12.16: 


A scalar continuous-time stochastic process, denoted by X[t,,tf) takes values x(t) € R for all real t,t; < 
t < ty, according to a probability that can be determined from the family of finite-dimensional pdfs. 
Px(ty),x(t2)y--.5x(ty) (X(t1), x(t2), ...,x(tn)) for all finite collections of ty, t, real and all possible positive integers 
n,ts<th < tp k=1,2,....n. 


The essential feature of this definition is the idea of a signal that is a function of the continuous time, f, 
and also random. The definition itself will not be used directly. Its main purpose is to allow the definitions 
of expectation and second-order process. 


12.3.2 Expectations 


As for discrete-time stochastic processes, the expectation, expected value, or mean of a scalar continuous- 
time stochastic process follows directly from Definitions 12.16 and 12.4. 


(oe) 


XPx(t)(x)dx. (12.59) 


A 
m(e) © BGC) = f 
Note that m/(t) is defined where X(t5,t/] is, that is, on the interval t, < t < tf. 
The covariance function is defined and computed for continuous-time stochastic processes in exactly 
the same way as for discrete-time stochastic processes. See Equations 12.6 and 12.7. 
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Definition 12.17: 


The covariance function of a scalar continuous-time stochastic process X{t,,1-] is denoted r(t, t) and is given by 


r(t, t) = E((x(t) — m(t))(x(t) — m(t)) 


om Tore) (12.60) 
= ff = meenyy— m(epainaco lary) de dy. 
—0O J—C 
As in the discrete-time case, it should be obvious from Equation 12.60 that 
r(t,t)=r(t,t) forallt,t, t¢<t, t<t (12.61) 
and 
r(t,t)>0O forallt, tt<t<t. (12.62) 


Because r(t, t) is a function of two variables and not a matrix, it is necessary to extend the idea of 
nonnegative definiteness to such functions in order to define and demonstrate this aspect of the “shape” 
of the covariance function. The idea behind the following definition is to form every possible symmetric 
matrix from time samples of r(t, t) and then to require that all those matrices be positive semidefinite in 
the usual sense. 


Definition 12.18: 


A real-valued function g(t,t); t, UE Rts < t,t < ty; is positive semidefinite if, for every finite collection 
ty, t2,...,ty3 t;<t < ty fori= 1,2,...,n and every real n-vector & = [01, O2,..., An]! 


n n 


SYS agoreg(te, te) = 0. (12.63) 


k-1 £-1 


The function g(t, t) is positive definite if strict inequality holds in Equation 12.63 whenever ty, t2,...; tn 
are distinct and a 4 0. 
It is easy to prove that an autocovariance function, r(t, t), must be positive semidefinite. First, 


Fe 2 
E (>: OK (X(t) — ms) =0 (12.64) 


k=1 


because the expectation of a perfect square must be > 0. Then, 


n 2 
O<E (>: ou ( (tk) — mts) 


k=1 


=e (>: » OL Ole (X (th) — ma( th) )(x(te) — mo) 


k=) fS1 


= S75 >) crgereE (x(t) — m(te))(x(te) — m(te))) 


keel £1 


n n 
= >. py apaer(ty, te). 


k=1 £=1 
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The definition of the characteristic function in the case of a continuous-time stochastic process is 
obvious from Equations 12.11, 12.59, and 12.60. Because it is generally a function of time as well as w it 
is not as useful in the continuous-time case. For this reason, it is not given here. 


12.3.3 Example: Continuous-Time Gaussian Stochastic Processes 


Definition 12.19: 


A scalar continuous-time stochastic process X{t,,1-] is a Gaussian process if the collection of random variables 
{x(t1), x(t2),...,X(tn)} is an n-vector Gaussian random variable for every finite set {t), t2,...,tnits <ti < 
tf,i=1,2,...,n}. 


It follows from Definition 12.8 in Section 12.2.3 that a Gaussian process is completely specified by its 
mean 
m(t) = E(x(t)) (12.65) 


and autocovariance 
r(t, t) = E((x(t) — m(t))(x(t) — m(t))). (12.66) 


Example 12.7: 


Wiener process (also known as Brownian motion): Let X[9,99) be a Gaussian process with 


m(t) = 0, 
r(t, t) = min(t, t). (12.67) 


Note that the interval of definition is infinite and open at the right. This is a very minor expansion 
of Definition 12.16. 

The Wiener process plays a fundamental role in the theory of stochastic differential equations (see 
Chapter 59); so it is worthwhile to derive one of its properties. Consider any ordered set of times 
to<t) <t2<---<tp. 

Define 

Vk = Xt — Xt —1 k=1,2,...,n. 


Then y;, is an increment in the process X[0,90). Consider the collection of increments 
y= {yy k =1,2,...,n}. 


Because X/9,90) is a Wiener process, each of the x;, is a Gaussian random variable (see Defini- 
tion 12.19). Therefore y,, the difference of Gaussian random variables, is also a Gaussian random 
variable for every k and y is an n-vector Gaussian random variable. Therefore, y is completely char- 
acterized by its mean and covariance. The mean is zero. The covariance is diagonal, as can be seen 
from the following calculation. 

Consider, for j < k — 1 (the means are zero), 


E(ygyj) = E((Xty — Xt, _ 1), — X4_1)) 
= E(xt, Xt;) — E(Kty Xt) — E(t, Mt _1) + EO) Xt_1) 
eh ie a aaa a a 
A similar calculation for j => k + 1 completes the demonstration that the covariance of y is diagonal. 
Because y is Gaussian, the fact that its covariance is diagonal proves that the y, k= 1,2,...,n are 


independent. This is a very important property of the Wiener process. It is usually described thus: the 
Wiener process has independent increments. 
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12.3.4 Stationarity, Ergodicity, and White Noise 


The concepts of stationarity and wide-sense stationarity for continuous-time stochastic processes are 
virtually identical to those for discrete-time stochastic processes. All that is needed is to define the 
continuous-time stochastic process on the infinite interval 


X(—00,00) = X(t) forall t,-00 <t <0 


and adjust the notation in Definitions 12.9 and 12.11. Only the latter is included here so as to emphasize 
the focus on second-order processes. 


Definition 12.20: 


A scalar continuous-time stochastic process, X(—o0,0)> is a second-order process if and only if E(x*(t)) < 00 
forallt,-—co <t<o. 


Definition 12.21: 


A second-order, continuous-time stochastic process X(_9,00) is wide-sense stationary if and only if its mean 
m(t) and covariance function r(t, t) satisfy 


m(t)=m, aconstant forallt, —oo<t<o, (12.68) 
r(t,t)=r(t+o,t+o) forallt,t,o, —oo<t,uo< om, (12.69) 


It is customary to define 


r(t)=r(t+t,t) forallt,t, —oo<t<oo. (12.70) 


The apparatus necessary to define continuous-time white noise is now in place. 


Definition 12.22: 


A scalar second-order continuous-time stochastic process, X(—0,90) is a white-noise process if and only if 


m(t) = E(x(t))=0 forallt, —oo<t<oo, (12.71) 
r(t)=rs(t), r>0, —co<t<o, (12.72) 


where 8(t) is the unit impulse (equivalently, the Dirac delta) function [5]. As in discrete time, the process is 
Gaussian white noise if, in addition, x(t) is Gaussian for all t. 


As in discrete time, it is often necessary to determine the properties of a stochastic process from 
a combination of prior knowledge and observations of one sample function. In order for this to be 
possible it is necessary that the process be stationary and that, in some form, averages over time of 
x(t), —0o < t < ow, converge to expectations. Mathematically, this can be expressed as follows. Given a 
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stationary continuous-time stochastic process X(_oo<t<o0), one wants 


lim = [ fend = E(f(x(0))) (12.73) 
t>oo 2t J_+ 


to hold for any reasonable function f(-). This is true, with some minor technical conditions for processes 
that are stationary and ergodic. See [4] for an introduction to this important but difficult subject. 


12.3.5 Transforms 


Again, in parallel to the discrete-time case, any transform that is useful in the analysis of continuous- 
time signals can be applied to continuous-time stochastic processes. The two obvious candidates are 
the Laplace and Fourier transforms [5]. The most important application is the completely deterministic 
Fourier transform of the autocovariance of a wide-sense stationary continuous-time stochastic process. 


Definition 12.23: 


Let X(_co,00) be a wide-sense stationary scalar continuous-time stochastic process with mean m and auto- 
covariance r(t). Assume es. [r(t)|dt < oo. Then the Fourier transform of the autocovariance function 


r(t), 
s(@) = i. r(the J dt (12.74) 


—0o 


is well defined and known as the spectral density of X(—90,00): 


The inverse of Equation 12.74 is the usual inverse of the Fourier transform. 


r(t) = a: [. s(w)el°! da. (12.75) 
27. 06 
When m = 0, _ 
(0) = E(x?(t)) = i s(0) 2. (12.76) 


IfX(_o0,90) is a voltage, current, or velocity, then r(0) can be interpreted as average power, at least to within 
a constant of proportionality. Then s(wo), where wo is any fixed frequency, must be the average power 
per unit frequency in X(_40,90) at the frequency wo [2]. This is why s(w) is called the spectral density. 


Example 12.8: 


SUPPOSE X(_¢0,00) is a white noise process (see Definition 12.24) with r = 1. The spectral density for 
this process is 


one) . 
s(w) = i a(the /°t dt = 1. 


—c 


Because all frequencies are equally present, as in white light, a process having s(w) = 1 is called a 
white-noise process. 


12.3.6 SISO Continuous-Time Linear Systems 


It is convenient to begin with linear systems described by their impulse response, h(t, t). The notation is 
shown in Figure 12.2. 
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Definition 12.24: 


The impulse response of a SISO continuous-time system is denoted by h(t, t), where h(t, t) is the value of 
the output at instant t when the input is a unit impulse, 8(t — t), at instant t. 


The response of linear system, S, to an arbitrary input, (4,00), is given by 


y(t) = in h(t, t)u(t)dt, —co<t<oo. (12.77) 


Because the second-order properties of y(_.o,.0) are completely described by its mean and autocovari- 
ance, it is worthwhile to know how to compute them from h(t,t) and the second-order properties of 
U(—oo,00): The trick is to compute them at each time, t. That is, 


m,(t) = E(y(t)) = E ( i i A(t, “)u(e)dr). 


By the linearity of integration 


= [- E(h(t, t)u(t))dt. 


Because h(t, t) is deterministic, 


= i A(t, t)E(u(t))dt. 


That is, 2 
my(t) = h(t, t)m,(t)dt, (12.78) 
—0o 
where " 
my,(t) = E(u(t)). 
Similarly, 


ryltst) = B(y(t) — my(£))(2) — my (2) 
=E ( i AGS OWE) eens / 


—o0O 


(oe) 


h(t, 02)(u(o2) — mu(o2))402 
=E (/ / h(t, 01 A(t, 62)(u(o1) — my(o1))(u(o2) — mu(o3))doidon) (12.79) 
7 i i, CACORE CCTM CIT Ce 


CO CO 
= i / CCN NCEE T ee 
=o. —C 


U(~co, 00) Y(~ce, c2) 


Input Output 


System 


FIGURE 12.2 Representation of a system with input u(_¢o,90) and output Y(_.5,0): 
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where 


ry(t, t) = E((u(t) — my(t))(a(e) — m,(1))). 


As in the discrete-time case, the Fourier transform can be used to simplify the calculations when the 
linear system is also time-invariant and u(_¢0,00) is wide-sense stationary. Denote the impulse response 
by h(t), where h(t) = h(t + t, t). Let m, = E(u(t)), and r,(t) = E((u(t + t) — my)(u(t) — m,)). It is then 
easy to show, by paralleling the argument leading to Equation 12.41, that 


My = hy (0)mu, (12.80) 


where 


lo) 
hy(@) = / h(t)e 4°* dt, the Fourier transform of h(t). 


—0oO 


Similarly, 


= 2 jot dw 
ry(t) = lhp(@)|"su(m)es" —, (12.81) 
Lae 2m 
where s,,(@) denotes the spectral density of u(_oo,90) (see Equation 12.74). This follows from virtually the 
same argument as Equation 12.42. By the uniqueness of Fourier transforms, Equation 12.81 implies 


sy(@) = |h¢(@)|7su(@). (12.82) 


It is necessary to define the notation for n-vector continuous-time stochastic processes before describing 
continuous-time systems in state-space form. Both of these are done in the following section. 


12.3.7 Vector Continuous-Time Stochastic Process and LTI Systems 


The vector case involves no new concepts but requires a more complicated notation. Because only second- 
order stochastic processes are discussed here all that is needed is a notation for the process, its mean, and 
its autocovariance. 

An n-vector continuous-time stochastic process, Xj), ),), is an n-vector random variable, x(t) = 
[x1 (t) xo(t)...X,(t)]/ at each instant of time t, t) < t < t). As throughout this chapter, vectors are denoted 
by lower-case underlined letters. Random variables and stochastic processes are bold letters. 

The mean of a vector stochastic process is defined at each instant of time; the autocovariance is defined 
for paired times. Thus, when x(t) is an n-vector random variable defined for all t,t) <t < fo (ie, X[t,1] 
is an n-vector stochastic process) 


m,(t)=E(x(t)) th <t<t (12.83) 


will denote its mean and 


R,(t, 0) = E((x(t) — m, (t))(&(t) — m,(t))') th <t, tSh (12.84) 


will denote its autocovariance. Note the transpose in Equation 12.84 and the capital R,. This empha- 
sizes that the autocovariance is an n x n matrix for each pair t and t. As in the discrete-time case, 
Equations 12.83 and 12.84 are evaluated element by element. Thus, 


my, (t) = E(x(t)) i=1,2,...,n, (12.85) 
Tq (ts T) = E((xi(t) — mx; (t)) (Kj (t) — mx; (t))). (12.86) 


Linear Systems and White Noise 12-21 


Example 12.9: 


Continuous-time n-vector white noise: Any white noise must be wide-sense stationary and have zero 
mean. Letting &(t) denote an n-vector continuous-time white noise at the instant t, 


0 forallt, -oo<t<o, (12.87) 


= E(&(t)) = 
= E&(t )e'(t) =0 forallt#At —co<t, t<@ (12.88) 


a (t, t) 
because the autocovariance of white noise must be equal to 0 except when t=t. At t=t, the 
autocovariance reduces to a covariance matrix (denoted by &, which must be positive semidefinite 
because it is a covariance matrix) times a unit impulse. A rigorous derivation of this is hard but the 
analogy to Equation 12.49 should be convincing. Thus, for all —oo < t,t < co 


R,(t, t) = E(t — 1), (12.89) 


where 3 is any positive semidefinite matrix. 

Now that vector continuous-time stochastic processes and vector white noise have been intro- 
duced it is possible to develop the state-space description of linear continuous-time systems driven 
by continuous-time stochastic processes. For convenience, only the case of LTI systems is described. 


Example 12.10: 


LTI system and white noise: Let xp be an n-dimensional second-order random variable with mean 
my, and covariance R,,. Leté,, and 910,00) be g- and p-dimensional wide-sense stationary second- 


order continuous-time Soeheic processes satisfying, for all t € [0, co) 


E(&(t)) = O and E(@(t)) = 0, (12.90) 
Re (t) = El (t + 8" (v)) = S8(0, (12.91) 
Ro(t) = E(@(t + 1)6'(t)) = O8(t), (12.92) 


where & and © are symmetric positive semidefinite matrices of appropriate dimension. 


Strictly speaking 5 i) and 0/9...) are not white noise processes because they are defined only in [0, oo). 
However, in the context of state-space analysis it is standard to call them white-noise processes. Obviously, 
what matters is not the name, but Equations 12.90 through 12.92. 

For convenience, assume that 


E((Xq — m,, )€ (t)) = 0, (12.93) 
E(x) — m,,)0/(t)) = 0, (12.94) 

and 
E(&(t)6'(t)) =0 forallt, t¢€[0,o). (12.95) 


The assumptions in Equations 12.93 through 12.95 remove some terms from the subsequent expres- 
sions, thereby saving space and the effort to compute them, but are otherwise inessential. 
Define the n- and p-vector random variables x(t) and y(t) by 


* = Ax(t) + Bu(t) + L&(t); x(0) =x, (12.96) 


y(t) = Cx(t) + 6(¢), (12.97) 


where u(t) is a deterministic m-vector, the control, and all the matrices are real, deterministic, and have 
the appropriate dimensions (that is, A is n x n, B is n x m, C is p x n, and L is n x q). The structure of 
this system is shown in Figure 12.3. 
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It follows from Equations 12.96 and 12.97 that x9...) and Yiojoo) are vector continuous-time stochastic 
processes. Assuming that u(t) is bounded for all t guarantees that x;, ty) and Yous) ate second-order for 
L0.t 
all finite ty. What are their second-order statistics? 
To compute the mean of x(t) take the expected value of both sides of Equation 12.96. That is, 


E(&(t)) = E(Ax(t) + Bu(t) + L&(t)). 


Interchanging the order of expectation and d/dt (both are linear; so their order can be reversed) and 
using linearity on the right-hand side of the equation gives 


“B(x(0) = AE(x()) + Bu(t); E(x(0)) = m,, (12.98) 


or 


m,(t) = Am,(t)+Bu(t); m,(0)= my, 


This is just a deterministic ordinary differential equation in state-space form for E(x(t)). Once E(x(t)) 
has been computed, taking expectation of both sides of Equation 12.97 gives 


E(y(t)) = CE(x(t)). (12.99) 


This is a deterministic algebraic equation. 
A derivation of the autocovariance of X10,1/) is beyond the scope of this chapter. A derivation can be 
found in [6, pp. 111-113]. 
The result is that 
R, (t,t) = E((x(t) — m,(1))(x(t) — m,(0))’) 


satisfies the differential equation 


R(t, t) = AR, (t,t) +R, (tA + LEL’ (12.100) 


with initial condition R,.(0,0) = Ry, and R,.(t, t), with t > t, satisfies the differential equation 


= AR (t,t) (12.101) 
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u(t) q = f x(t 4 (+) 
a B 
Control ag Ko Measured 
output 


A == 


FIGURE 12.3 Block diagram of the LTI system described by Equations 12.96 and 12.97. 
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while R,.(t, f), again with t > 1, satisfies 


OR, (1, f) 


=R,(t, tA’. 12.102 
= Ryo EA (12.102) 


The initial condition for both Equations 12.102 and 12.103 is R,(t, t), computed from Equation 12.100. 


12.4 Conclusions 


The second-order properties of many zero-mean stochastic processes can be reproduced by an LTI system 
with white-noise input. The precise result for scalar continuous-time systems is the following [2]. Let s(w) 
be a real nonnegative absolutely integrable function such that 


8: (] 
/ Pe do < 00. (12.103) 
£65 w 


Then there exists a square integrable function hy(m), —co < w < 00, such that 


|hp(w)|? = s(w); (12.104) 
ee dw 
h(t) = / hy (we? — =0 fort <0, (12.105) 
= 66) 2m 
oo . r 
hy (o + jw) = / elt) h(t)dt £0, w > 0, (12.106) 
=O0 


Because the spectral density of white noise is one for all w, Equation 12.104 means that the spectral 
density, s(w), is identical to the spectral density of the output of an LTI system with impulse response 
h(t) driven by a white-noise input. Equation 12.105 means that h(t) is causal. Equation 12.106 means 
that the transfer function hy() is minimum phase. Similar results hold in discrete time and for vector 
signals [7,8]. 

The restriction to second-order processes with zero mean is unimportant because the mean is deter- 
ministic. Thus, the only significant limitation on the use of a linear system driven by white noise as a 
model for second-order stochastic processes is Equation 12.103. The major limitation of the model is that 
the second-order statistics are not always a good description of a stochastic process. The second-order 
statistics are most useful when the process is approximately Gaussian. Processes that are very different 
from Gaussian, especially in the vicinity of the mean, are not usually well modeled by their second-order 
statistics. 


12.5 Notation 


Lower case letters such as x, y, and r denote real, deterministic scalars. The same letters when underlined 
denote vectors. 

Lower case letters i, j,k, 2, m, and n denote integers (j is also used to indicate /—1 and m for mean—the 
meaning should be evident from the context). 

Upper case, underlined, capital letters such as A, B, C, L, ©, and & denote real deterministic matrices. 

Lower case, bold letters such as x and y denote scalar random variables. The same letters when 


underlined denote vector random variables (or discrete-time stochastic processes). 

Upper case, underlined bold letters such as X, Y,@, and & denote vector discrete-time stochastic 
processes. 

Lower case, bold letters with interval subscripts such as x(_¢,90) denote stochastic processes. 
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13.1 Introduction 


The purpose of this chapter is to provide an overview of Kalman filtering concepts. We cover only a 
small portion of the material associated with Kalman filters. Our choice of material is primarily motivated 
by the material needed in the design and analysis of multivariable feedback control systems for linear 
time-invariant (LTI) systems and, more specifically, by the design philosophy commonly called the linear 
quadratic Gaussian (LQG) method. Thus, from a technical perspective, we cover, without proofs, the 
so-called steady-state, constant-gain LTI Kalman filters. 

More details regarding different formulations and solutions of Kalman filtering problems (continuous- 
time linear time-varying, discrete-time linear time-varying, and time-invariant) can be found in several 
classic and standard textbooks on the subject of estimation theory. Please see Further Reading at the end 
of this chapter. 

The steady-state constant-gain Kalman filter is an algorithm that is used to estimate the state variables 
of a continuous-time LTI system, subject to stochastic disturbances, on the basis of noisy measurements 
of certain output variables. Thus, the Kalman filtering algorithm combines the information regarding 
the plant dynamics, the probabilistic information regarding the stochastic disturbances that influence the 
plant state variables, as well as that regarding the measurement noise that corrupts the sensor measure- 
ments, and the deterministic controls. 

Credit for “inventing” this algorithm is usually given to Dr. Rudolph E. Kalman, who presented the key 
ideas in the late 1950s and early 1960s. Although the Kalman filter is an alternative representation of the 
Wiener filter [1], Kalman’s contribution was to tie the state estimation problem to the state-space models, 
and the (then new) concepts of controllability and observability [2-5]. Thousands of papers have been 
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written about the Kalman filter and its numerous applications to navigation, tracking, and estimator and 
controller design in defense and industrial applications. 


13.2 Problem Definition 


In this section we present the definition of the basic stochastic estimation problem for which the Kalman 
filter (KF) yields an “optimal” solution. First we present the plant state dynamics. We deal with a finite- 
dimensional LTI system whose state vector x(t), x(t) € R” (a real n-vector), obeys the stochastic differen- 
tial equation 


X(t) = Ax(t) + Bu(t) + LE(t) (13.1) 


where u(t), u(t) € R™ is the deterministic control vector (assumed known), &(t), &(t) € R4 is a vector- 
valued stochastic process, often called the process noise, that acts as a disturbance to the plant dynamics. 
Vectors are underlined, lower case letters and matrices are underlined, upper case letters. The process 
noise &(t) is assumed to have certain statistical properties, corresponding to a stationary (time-invariant) 
continuous-time white Gaussian noise with zero mean, i.e., 


E{&(t)} =0, forallt (13.2) 
and its covariance matrix is defined by 


cov [&(t); &(t)] = E{&(£)§'(t)} = B8(t — 1) (13.3) 


with 8(t — t) being the Dirac delta function (impulse at t = t). The matrix © is called the intensity matrix 
of &(t) and it is a symmetric positive definite matrix, ie., 


es) 
ll 
[ey 
V 
oO 


(13.4) 


Remark 13.1 


Continuous-time white noise does not exist in nature; it is the limit of a broadband stochastic process. 
In the frequency domain, continuous-time white noise corresponds to a stochastic process with constant 
power spectral density as a function of frequency. This implies that continuous-time white noise has con- 
stant power at all frequencies, and therefore has infinite energy! White noise is completely unpredictable, 
as can be seen from Equation 13.3 because it is uncorrelated for any t 4 t, while it has finite variance 
and standard deviation at t = t. This is obviously an approximation to reality. As with the Dirac delta 
function, 8(t), white noise creates some subtle mathematical issues, but is nonetheless extremely useful in 
engineering. 


Remark 13.2 


The state x(t), the solution of Equation 13.1, is a well-defined physical stochastic process, a so-called 
colored Gaussian random process, and it has finite energy. Its power spectral density rolls off at high 
frequencies. 

Next, we turn our attention to the measurement equation. We assume that our sensors cannot directly 
measure all of the physical state variables, the components of the vector x(t) of the plant given in 
Equation 13.1. Rather, in the classical Kalman filter formulation we assume that we can measure only 
certain output variables (linear combinations of the state variables) in the presence of additive continuous- 
time white noise. 
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The mathematical model of the measurement process is as follows: 
y(t) = Cx(t) + 0(f) (13.5) 
The vector y(t) € R? represents the sensor measurement. The measurement or sensor noise 0(t) € RP 


is assumed to be a continuous-time white Gaussian random process, independent of &(t), with zero 
mean, Le., 


E{@(t)}=0, for allt (13.6) 


and covariance matrix 


cov [0(t); 8(t)] = E{9(t)0'(t)} = ©O8(t — t) (13.7) 


where the sensor noise intensity matrix is symmetric and positive definite, i.e., 


© 


=0'/>0 (13.8) 


Figure 13.1 shows a visualization, in block diagram form, of Equations 13.1 and 13.5. 


13.2.1 The State Estimation Problem 


Imagine that we have been observing the control u(t) and the output y(t) over the infinite past up to the 
present time ¢t. Let 


U(t) = {u(t); 00 < t St} (13.9) 
Y(t) = {y(t); —00 <t<t} (13.10) 


denote the past histories of the control and output, respectively. 

The state estimation problem is as follows: given U(t) and Y(t), find a vector x(t), at time t, which is 
an “optimal” estimate of the present state x(t) of the system defined by Equation 13.1. 

Under the stated assumptions regarding the Gaussian nature of &(t) and 0(t) the “optimal” state 
estimate is the same for an extremely large class of optimality criteria [6]. This generally optimal estimate 
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FIGURE 13.1 A stochastic linear dynamic system. 
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is the conditional mean of the state, i-e., 
&(1) 2 E{x(1)|U(), YO) (13.11) 
One can relax the Gaussian assumption and define the optimality of the state estimate x(t) in different 


ways. One popular way is to demand that x(t) be generated by a linear transformation on the past “data” 
U(t) and Y(t), such that the state estimation error x(t) 


x 2 x(t) —2(0) (13.12) 
has zero mean, i.e., 
E{x(t)} = 0 (13.13) 
and the cost functional 
J=E (>: s0o| = E {x'(t)x(t)} = tr[E{x(0)x'(}] (13.14) 
i=1 


is minimized. 
The cost functional J has the physical interpretation that it is the sum of the error variances E{x?(t)} 
for each state variable. If we let & denote the covariance matrix (stationary) of the state estimation error 


ES ERO (1) (13.15) 

then the cost, J, of Equation 13.14 can also be written as 
J =tr[X] (13.16) 
Bottom Line: We need an algorithm that translates the signals we can observe, u(t) and y(t), into a state 


estimate X(t), such that the state estimation error X is “small” in some well-defined sense. The KF is the 
algorithm that does just that! 


13.3 Summary of the Kalman Filter Equations 


13.3.1 Additional Assumptions 
In this section we summarize the on-line and off-line equations that define the Kalman filter. Before we 


do that we make two additional “mild” assumptions 


[A, L] is stabilizable (or controllable) (13.17) 
[A, C] is detectable (or observable) (13.18) 


[A, L] is controllable, means that the process noise E(t) excites all modes of the system defined by 
Equation 13.1; [A, C] is observable means that the “noiseless” output y(t) = Cx(f) contains information 
about all state variables. If [A, L] is stabilizable, the modes of the system that are not excited by &(t) are 
asymptotically stable; if [A, C] is detectable, the unobserved modes are asymptotically stable. 


13.3.2 The Kalman Filter Dynamics 


The function of the KF is to generate in real time the state estimate x(t) of the state x(t). The KF is 
actually an LTI dynamic system, of identical order (1) to the plant Equation 13.1, and is driven by (1) the 
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deterministic control input u(t), and (2) the measured output vector y(t). The Kalman filter dynamics 
are given as follows: 


dx(t) 
dt 


= Ax(t)+ Bu(t)4 Hly(t) Cx(t)] (13.19) 


A block diagram visualization of Equation 13.19 is shown in Figure 13.2. Note that in Equation 13.19 
all variables have been defined previously, except for the KF gain matrix H, whose calculation is carried 
out off-line and is discussed shortly. 

The filter gain matrix H multiplies the so-called residual or innovations vector 


r(t) = y(t) — Cx(t) (13.20) 


and updates the time rate of change, dx(t) /dt, of the state estimate x(t). The residual r(f) is like an “error” 
between the measured output y(t), and the predicted output Cx(t). 


Remark 13.3 


From an intuitive point of view the KF, defined by Equation 13.19 and illustrated in Figure 13.2, can 
be thought as a model-based observer or state reconstructor. The reader should carefully compare the 
structures depicted in Figures 13.1 and 13.2. The plant/sensor properties, reflected by the matrices A, B, 
and C, are duplicated in the KF*. The state estimate x(t) is continuously updated by the actual sensor 
measurements, through the formation of the residual r(t) and the “closing” of the loop with the filter gain 
matrix H. 

The KF dynamics of Equation 13.19 can also be written in the form 


dx(t) 
dt 


= [A — HC]&(t) + Bu(t) + Hy(t) (13.21) 


From the structure of Equation 13.21 we can immediately see that the stability of the KF is governed by 
the matrix A — HC. At this point of our development we remark that the assumption in Equation 13.18, 
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FIGURE 13.2 The structure of the Kalman Filter. The control, u(t), and measured output, y(t), are those associated 
with the stochastic system of Figure 13.1. The filter gains matrix H is computed in a special way. 


* No signal corresponding to L&(t) shows up in Figure 13.2. This is because we assumed that §(t) had zero-mean and was 


completely unpredictable. Thus, the best estimate for §(¢) given data up to time f is 0. 
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ice., the detectability of [A, C], guarantees the existence of at least one filter gain matrix H such that the 
KF is stable, i-e., 
Rex[A—HC]<0; i=1,2,...,n (13.22) 


where },; is the ith eigenvalue of [A — HC]. 


13.3.3 Properties of Model-Based Observers 


We have remarked that the KF gain matrix H is calculated in a very special way. However, it is extremely 
useful to examine the structure of Equations 13.19 or 13.21 and Figure 13.2 with a filter gain matrix H 
that is arbitrary except for the requirement that Equation 13.22 holds. Thus, for the development that 
follows in this subsection think of H as being a fixed matrix. 

As before let x(t) denote the state estimation error vector 


&(t) 2 x(t) — X(t) (13.23) 
It follows that 


dx(t) dx(t) dx(t) 
dt ~—s dt dt 
Next, we substitute Equations 13.1, 13.5, and 13.21 into Equation 13.24 and use Equation 13.23 as 
appropriate. After some easy algebraic manipulations we obtain the following stochastic vector differential 
equation for the state estimation error x(t): 


dx(t) 
dt 
Note that, in view of Equation 13.22, the estimation error dynamic system is stable. Also note that the 
deterministic signal Bu(t) does not appear in the error Equation 13.25. 
Under our assumptions that the system is stable and was started at the indefinite past (to > —0o), it is 
easy to verify that 


(13.24) 


= [A — HC]&(t) + Lé(t) — HO(t) (13.25) 


E{x(t)} =0 (13.26) 


This implies that any stable model-based estimator of the form shown in Figure 13.2, with any filter gain 
matrix H, gives us unbiased (that is, E(x(t)) = E(x(t)) estimates. 

Using next elementary facts from stochastic linear system theory one can calculate the error covariance 
matrix » of the state estimation error x(t) 


E> cov{X(t)sX()] = ERX ()} (13.27) 
The matrix & is the solution of the so-called Lyapunov matrix equation (linear in X) 


[A — HC] =+ X(A— HC) + LEL’ + HOH’ =0 (13.28) 


with 
TH 5S (13.29) 


Thus, for any given filter gain matrix H we can calculate* the associated error covariance matrix © 
from Equation 13.28. Recalling Equation 13.16, we can evaluate, for a given H, the quality of the estimator 
by calculating 

J=t[d] (13.30) 


The specific way that the KF gain is calculated is by solving a constrained static deterministic optimiza- 
tion problem. Minimize Equation 13.30 with respect to the elements hj; of the matrix H subject to the 
algebraic constraints given in Equations 13.28 and 13.29. 


* The MATLAB® and MATRIXx!™ software packages can solve Lyapunov equations. 
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13.3.4 The Kalman Filter Gain and Associated Filter Algebraic Riccati 
Equation (FARE) 


We now summarize the off-line calculations that define fully the Kalman filter (Equation 13.19 or 13.21). 
The KF gain matrix H is computed by 


H=xC’o"! (13.31) 


where » is the unique, symmetric, and at least positive semidefinite solution matrix of the so-called filter 
algebraic Riccati equation (FARE) 


0=AD+ ZA’ +LEL’ —xC’@"!cz (13.32) 


with 
y=’ 0 (13.33) 


Remark 13.4 
The formula for the KF gain can be obtained by setting 


a 
ahi 


tr[X] =0 (13.34) 


where & is given by Equation 13.28. The result is Equation 13.31. Substituting Equation 13.31 into 
Equation 13.28 one deduces the FARE (Equation 13.32). 


13.3.5 Duality between the KF and LQ Problems 


The mathematical problems associated with the solution of the LQ and KF are dual. This duality was 
recognized by R.E. Kalman as early as 1960 [2]. 

The duality can be used to deduce several properties of the KF simply by “dualizing” the results of the 
LQ problem. A summary of the KF properties is given in Section 13.4. 


13.4 Kalman Filter Properties 


13.4.1 Introduction 


In this section we summarize the key properties of the Kalman filter. These properties are the “dual” of 
those for the LQ controller. 


13.4.2 Guaranteed Stability 
Recall that the KF algorithm is 


ao = [A— HCIK(t) + Bult) + Hy(#) (13.35) 


Then, under the assumptions of Section 35.3, the matrix [A — HC] is strictly stable, i-e., 


Redj[A-HC] <0; i=1,2,...,n (13.36) 
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13.4.3 Frequency Domain Equality 


One can readily derive a frequency domain equality for the KF. In the development that follows, let 


69] 


=I (13.37) 


Let us make the following definitions: let Gx-(s) denote the KF loop-transfer matrix 


Gents) = C(sl — A) 1H (13.38) 


GH, (s) 2 H'(-sI — A’)"!C’ (13.39) 


where [A]! denotes the complex conjugate of the transpose of an arbitrary complex matrix A. Let Gro, (s) 
denote the filter open-loop transfer matrix (from §(t) to y(t)) 


Groy(s) & C(sl — A) IL (13.40) 
GHoz(s) = L(-sl— A’ (13.41) 

Then the following equality holds 
[1+ Gxp(s)]OU+ Gxp(s)}" = 0+ Gro, (s)GFor (s) (13.42) 

If 
O=pl p>o (13.43) 
then Equation 13.42 reduces to 
1 

[I+ Gep(s) IL + Grp (s)# =I+ yp oFOL (s)\GE, (s) (13.44) 


13.4.4 Guaranteed Robust Properties 


The KF enjoys the same type of robustness properties as the LQ regulator. The following properties are 
valid if 
© = diagonal matrix (13.45) 


From the frequency domain equality (Equation 13.42) we deduce the inequality 


(1+ Grr(s)ILE + Ggp(s)}? > 1 (13.46) 


From the definition of singular values we then deduce that 


OminLI + Ggp(s)]=1 or Omaxll + Gep(s)]* <1 (13.47) 


= 1 2 
Ominll + Gyp(s)] = 5 OF Omaxll + Gxr(s)] "Gxp <2 (13.48) 


13.5 The Accurate Measurement Kalman Filter 


We summarize the properties of the Kalman Filter (KF) problem when the intensity of the sensor noise 
approaches zero. In a mathematical sense this is the “dual” of the so-called “cheap-control” LQR problem. 
The results are fundamental to the loop transfer recovery (LTR) method applied at the plant input. 
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13.5.1 Problem Definition 


Consider as before, the stochastic LTI system 


X(t) = Ax(t) + LE(t) (13.49) 
y(t) = Cx(t) + O(t) (13.50) 


We assume that the process noise §(t) is white, zero-mean, and with unit intensity, i-e., 
E{gé(t)§'(t) = 18(t — 1) (13.51) 


We also assume that the measurement noise 0(f) is white, zero-mean, and with intensity indexed by 1, 
that is, 
E{9(t)0'(t) = p1S(t — 1) (13.52) 


Definition 13.1: 


The accurate measurement KF problem is defined by the limiting case 
u—> 0 (13.53) 


corresponding to essentially noiseless measurements. 


Under the assumptions that [A, L] is stabilizable and that [A, C] is detectable we know that the KF is a 
stable system and generates the state estimates X(t) by 


dx(t) 
dt 


=[A-H,,Cl&(t) +H, y(t) (13.54) 


where we use the subscript 1 to stress the dependence of the KF gain matrix H,, upon the parameter |. 
We recall that H,, is computed by 
1 


H, =-z,C’ (13.55) 


where the error covariance matrix %,,, also dependent upon 1, is calculated by the solution of the FARE: 


1 
/ / / 
O=AE, +2E,A' +LL'——E,CCE, (13.56) 
We seek insight about the limiting behavior of both xy and H, past > 0. 


13.5.2 The Main Result 


In this section we summarize the main result in terms of a theorem. 


Theorem 13.1: 


Suppose that the transfer function matrix from the white noise &(t) to the output y(t) for the system defined 
by Equations 13.49 and 13.50, i.e., the transfer function matrix 7 
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W(s) * C(sl —A)“!L (13.57) 
is minimum phase. Then, 
lim X, = 0 (13.58) 
p—>0 
and 
lim Jee, =LW; WWe=l (13.59) 
p> 0 


Proof. This is theorem 4.14 in Kwakernaak and Sivan [7], pp. 370-371. 


Remark 13.5 


It can be shown that the requirement that W(s), given by Equation 13.57, be minimum phase is both a 
necessary and sufficient condition for the limiting properties given by Equations 13.58 and 13.59. 


Remark 13.6 


The implication of Equation 13.58 is that, in the case of exact measurements upon a minimum phase 
plant, the KF yields exact state estimates, since the error covariance matrix is zero. This assumes that 
the KF has been operating upon the data for a sufficiently long time so that initial transient errors have 
died out. 


Remark 13.7 

For a non-minimum phase plant 
lim D, #0 (13.60) 
pu>0 


Hence, perfect state estimation is impossible for non-minimum phase plants. 


Remark 13.8 
The limiting behavior (with L = B) of the Kalman Filter gain 


lim Jedi, =BW; W’W=I (13.61) 
—>0 


is the precise dual of the limiting behavior of the LQ control gain 


lim /pG,=WC; WW=I (13.62) 
p—>0 


for the minimum phase plant 
G(s) = C(sl — A)"'B (13.63) 


The relation (Equation 13.61) has been used by Doyle and Stein [8] to apply the LTR method at the plant 
input, while Equation 13.62 has been used by Kwakernaak [9] to apply the LTR method at the plant 
output (see also Kwakernaak and Sivan [7], pp. 419-427). 
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14.1 Introduction 
An ordinary differential equation of the form 

x(t) +f (f)x(t) — b(t)x?(t) + c(t) =0 (14.1) 


is known as a Riccati equation, deriving its name from Jacopo Francesco, Count Riccati (1676-1754) 
[1,2], who studied a particular case of this equation from 1719 to 1724. 

For several reasons, a differential equation of the form of Equation 14.1, and generalizations thereof 
comprise a highly significant class of nonlinear ordinary differential equations. First, they are intimately 
related to ordinary linear homogeneous differential equations of the second order. Second, the solutions 
of Equation 14.1 possess a very particular structure in that the general solution is a fractional linear 
function in the constant of integration. In applications, Riccati differential equations appear in the classical 
problems of the calculus of variations and in the associated disciplines of optimal control and filtering. 

The matrix Riccati differential equation refers to the equation [3-5] 


X(t) + X(t)A(t) — D(t)X(t) — X(t)B(t)X(t) + C(t) = 0 (14.2) 


defined on the vector space of real m x n matrices. Here, A, B,C, and D are real matrix functions of 
the appropriate dimensions. Of particular interest are the matrix Riccati equations that arise in optimal 
control and filtering problems and that enjoy certain symmetry properties. This chapter is concerned 


14-1 


14-2 Control System Advanced Methods 


with these symmetric matrix Riccati differential equations and concentrates on the following four major 
topics: 


¢ Basic properties of the solutions 

« Existence and properties of constant solutions 

« Asymptotic behavior of the solutions 

¢ Methods for the numerical solution of the Riccati equations 


14.2 Optimal Control and Filtering: Motivation 


The following problems of optimal control and filtering are of great engineering importance and serve to 
motivate our study of the Riccati equations. 
A linear-quadratic optimal control problem [6] consists of the following. Given a linear system 


x(t) = Fx(t)+ Gu(t), x(to)=c, y(t) = Hx(t), (14.3) 


where x is the n-vector state, u is the q-vector control input, y is the p-vector of regulated variables, and 
F, G, H are constant real matrices of the appropriate dimensions. One seeks to determine a control input 
function u over some fixed time interval [t,, t2] such that a given quadratic cost functional of the form 


to 
n(t, fT) = i Ly’ (t)y(t) + uw (t)u(t)] dt + x’ (t)Tx(tr), (14.4) 
ty 


with T being a constant real symmetric (T = T’) and nonnegative definite (T > 0) matrix, is afforded a 
minimum in the class of all solutions of Equation 14.3, for any initial state c. 
A unique optimal control exists for all finite t2 — t; > 0 and has the form 


u(t) = —G'P(t, t, T)x(t), 
where P(t, tz, T) is the solution of the matrix Riccati differential equation 
—P(t) = P(t)F + F’ P(t) — P(t)GG' P(t) + HH (14.5) 


subject to the terminal condition 
P (t2) =T. 


The optimal control is a linear state feedback, which gives rise to the closed-loop system 
X(t) = [F — GG'P(t, tp, T) x(t) 
and yields the minimum cost 
1*(t, ta, T) = ¢' P(t, tr, T)e. (14.6) 
A Gaussian optimal filtering problem [7] consists of the following. Given the p-vector random process 
z modeled by the equations 
x(t) = Fx(t) + Gv(t), 


(14.7) 
z(t) = Hx(t)+ w(t), 


where x is the n-vector state and v, w are independent Gaussian white random processes (respectively, 
q-vector and p-vector) with zero means and identity covariance matrices. The matrices F,G, and H are 
constant real ones of the appropriate dimensions. 
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Given known values of z over some fixed time interval [f,, t2] and assuming that x(t,) is a Gaussian 
random vector, independent of v and w, with zero mean and covariance matrix S, one seeks to determine 
an estimate x(t2) of x(t2) such that the variance 


0(S, ty, t2) = Ef [x(t2) — X(t) Ix(tr) — X(b)I'F (14.8) 


of the error encountered in estimating any real-valued linear function f of x(tz) is minimized. 
A unique optimal estimate exists for all finite tf; — t; > 0 and is generated by a linear system of the form 


X(t) = FX(t) + Q(S, ti, t)H’e(t), X(to)=0, e(t) = z(t) — Hx(t), 


where Q(S, t1, f) is the solution of the matrix Riccati differential equation 


Q(t) = QU)F’ + FQ(t) — Q(t)H’HQ(t) + GG (14.9) 
subject to the initial condition 
Q(t) =S. 
The minimum error variance is given by 
o*(S, tb) =f' QS, ty b)f. (14.10) 


Equations 14.5 and 14.9 are special cases of the matrix Riccati differential equation 14.2 in that A, B, C, 
and D are constant real n x n matrices such that 


B=B, C=C, D=-A’. 


Therefore, symmetric solutions X(t) are obtained in the optimal control and filtering problems. 
We observe that the control Equation 14.5 is solved backward in time, while the filtering Equation 14.9 
is solved forward in time. We also observe that the two equations are dual to each other in the sense that 


P(t, bh, T) — Q(s, hh, t) 


on replacing F, G, H, T, and tz — t in Equation 14.5 respectively, by F’, H’, G’, S, and t — t or, vice versa, 
on replacing F, G, H, S, and t — t in Equation 14.9 respectively, by F’, H’, G’, T, and ty — t. This makes it 
possible to dispense with both cases by considering only one prototype equation. 


14.3 Riccati Differential Equation 


This section is concerned with the basic properties of the prototype matrix Riccati differential equation 
X(t) +. X(H)A+A’X(t) — X(t)BX(t) + C =0, (14.11) 


where A, B, and C are constant real n x n matrices with B and C being symmetric and nonnegative 


definite, 
B=B,, B>0 and C=C’, C>0. (14.12) 


By definition, a solution of Equation 14.11 is a real n x n matrix function X(t) that is absolutely 
continuous and satisfies Equation 14.11 for f on an interval on the real line R. 

Generally, solutions of Riccati differential equations exist only locally. There is a phenomenon called 
finite escape time: the equation 


X(t) = x7(t) +1 


has a solution x(t) = tant in the interval (— 2, 0) that cannot be extended to include the point t = ak 


However, Equation 14.11 with the sign-definite coefficients as shown in Equation14.12 does have global 
solutions. 
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Let X(t, t2, T) denote the solution of Equation 14.11 that passes through a constant real n x n matrix 
T at time t). We shall assume that 
T=T and T>0. (14.13) 


Then the solution exists on every finite subinterval of R, is symmetric, nonnegative definite and enjoys 
certain monotone properties. 


Theorem 14.1: 


Under the assumptions of Equations 14.12 and 14.13, Equation 14.11 has a unique solution X(t, to, T) 
satisfying 
X(t,t.T)=X"(t,,T), X(tb,T) > 0 


for every T and every finite t, tz, such that t > to. 


This can most easily be seen by associating Equation 14.11 with the optimal control problem described 
in Equations 14.3 through 14.6. Indeed, using Equation 14.12, one can write B= GG’ and C = H’H for 
some real matrices G and H. The quadratic cost functional n of Equation 14.4 exists and is nonnegative 
for every T satisfying Equation 14.13 and for every finite t. — t. Using Equation 14.6, the quadratic form 
c/X(t, t2, T)c can be interpreted as a particular value of 1 for every real vector c. 

A further consequence of Equations 14.4 and 14.6 follows. 


Theorem 14.2: 


For every finite t,, tg and t1, t2 such that t) < 11 < 12 < ta, 


X (ty, T, 0) < X(t}, T2, 0) 
X(t2; f2,0) < X(11, f2,0) 


and for every T, < To, 
X(t, 2, Ti) < X(t, te, Ta). 


Thus, the solution of Equation 14.11 passing through T = 0 does not decrease as the length of the 
interval increases, and the solution passing through a larger T dominates that passing through a smaller T. 

The Riccati Equation 14.11 is related in a very particular manner with linear Hamiltonian systems of 
differential equations. 


Theorem 14.3: 


Let 


C04) & a 


P21 Px 


be the fundamental matrix solution of the linear Hamiltonian matrix differential system 


U(t)| [A  —B][U() 
V(t)| |—-C —A'}| V(t) 
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that satisfies the transversality condition 
V(t2) = TU(t2). 
If the matrix ® 1, + ®12T is nonsingular on an interval [t, tz], then 
X(t, to, T) = (®21 + Ba2T (P11 + OT) | (14.14) 
is a solution of the Riccati Equation 14.11. 


Thus, if V(t2) = TU(t2), then V(t) = X(t, t2, T)U(t) and the formula of Equation 14.14 follows. 
Let us illustrate this with a simple example. The Riccati equation 


x(t) =22(t)—1, x(0) =T 


satisfies the hypotheses of Equations 14.12 and 14.13. The associated linear Hamiltonian system of 
equations reads 

u(t)| | 0 1} } ule) 

v(t)} |—-1 0 |} v(t) 


u(t)| | cosht —sinht}] | u(0) 
v(t)} |—sinht  cosht || v(0)]’ 


where v(0) = Tu(0). Then the Riccati equation has the solution 


and has the solution 


—sinh t+ T cosht 


t,0, 7) = ————__—_—_ 
a ) cosh t — T sinh t 


for all t < 0. The monotone properties of the solution are best seen in Figure 14.1. 


FIGURE 14.1 Graph of solutions. 
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14.4 Riccati Algebraic Equation 


The constant solutions of Equation 14.11 are just the solutions of the quadratic equation 
XA+A’X —XBX+C=0, (14.15) 


called the algebraic Riccati equation. This equation can have real n x n matrix solutions X that are 
symmetric or nonsymmetric, sign definite or indefinite, and the set of solutions can be either finite or 
infinite. These solutions will be studied under the standing assumption of Equation 14.12, namely 


B=B', B>0 and C=C, C=0. 


14.4.1 General Solutions 


The solution set of Equation 14.15 corresponds to a certain class of n-dimensional invariant subspaces of 
the associated 2n x 2n matrix 


as | A ei (14.16) 


This matrix has the Hamiltonian property 


[A altel) 


It follows that H is similar to —H’ and therefore, the spectrum of H is symmetrical with respect to the 
imaginary axis. 
Now suppose that X is a solution of Equation 14.15. Then 


uff] = [FJ a—nn. 


Denote J = U~!(A — BX)U, the Jordan form of A — BX and put V = XU. Then 


which shows that the columns of 


are Jordan chains for H, that is, sets of vectors x, x2,...,x, such that x, 4 0 and for some eigenvalue 
of H 


Ax, = dx} 
Axj = xj + Xj41, pS 2d) e5e 


In particular, x; is an eigenvector of H. Thus, we have the following result. 


Theorem 14.4: 


Equation 14.15 has a solution X if and only if there is a set of vectors x1,X2,...;Xn forming a set of Jordan 
chains for H and if 
xi = Ey > 
Vi 


where uj is an n-vector, then uy, U2,..., Uy are linearly independent. 
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Furthermore, if 
U=[u,... un], V=[y,...Vn], 


every solution of Equation 14.15 has the form X = VU—! for some set of Jordan chains x1, xX2,... Xn for H. 
To illustrate, consider the scalar equation 
X* + pX+q=0, 


where p, q are real numbers and q < 0. The Hamiltonian matrix 


-f =| 
2 
has eigenvalues } and —, where 
pee 2 
ee Gi A 


X,= th, X= h. 
aos = e 
If } = 0 there exists one Jordan chain, 
1 0 
=!) Pl, m= | > 
ae 1 
2 
which yields the unique solution 
P 
X;=--. 
oe 


Theorem 14.4 suggests that, generically, the number of solutions of Equation 14.15 to be expected will 
2 
not exceed the binomial coefficient ( , the number of ways in which the vectors x1, x2,...,X, can be 


chosen from a basis of 2n eigenvectors for H. The solution set is infinite if there is a continuous family of 
Jordan chains. To illustrate this point consider Equation 14.15 with 


+=[5 al 2a[o al e=[o ol: 


The Hamiltonian matrix 


0 0 -1 0O 
0 0 0 -!1 
is 0 0 O 0 
0 0 O 0 
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has the eigenvalue 0, associated with two Jordan chains 


a Cc c a 
xy= p > O= ? » and 3= a x: X= : , 
0 —a 0 —c 
0 —b 0 —d 


where a, b and c,d are real numbers such that ad — bc = 1. The solution set of Equation 14.15 consists of 
the matrix 


and two continuous families of matrices 


ab —a? 


—cd 2 
Xl) =[ fp = and Xadlod =| a 


Having in mind the applications in optimal control and filtering, we shall be concerned with the 
solutions of Equation 14.15 that are symmetric and nonnegative definite. 


14.4.2 Symmetric Solutions 


In view of Theorem 14.4, each solution X of Equation 14.15 gives rise to a factorization of the characteristic 
polynomial x 4 of H as 
xa(s) = (—1)"q(s)qu(s), 


where q = x a_sx. If the solution is symmetric, X = X’, then qi(s) = q(—s). This follows from 


Por aA BP op (A=Be =B 
E i fe a E “|-[ 0 a weil 


There are two symmetric solutions that are of particular importance. They correspond to a factorization 
XH(s) = (—1)"q(s)q(-s) 


in which q has all its roots with nonpositive real part; it follows that q(—s) has all its roots with nonnegative 
real part. We shall designate these solutions X; and X_. 

One of the basic results concerns the existence of these particular solutions [8]. To state the result, we 
recall some terminology. A pair of real n x n matrices (A, B) is said to be controllable (stabilizable) if the 
nx 2n matrix [\J—A_ B] has linearly independent rows for every complex i (respectively, for every 
complex > such that Re } > 0). The numbers \ for which [XJ—A_ B] loses rank are the eigenvalues 
of A that are not controllable (stabilizable) from B. A pair of real n x n matrices (A, C) is said to be 


observable (detectable) if the 2n x n matrix ie a 


C ‘| has linearly independent columns for every complex 


d (respectively, for every complex i such that Re \ > 0). The numbers > for which Pe ‘| loses rank 


are the eigenvalues of A that are not observable (detectable) in C. Finally, let dim V denote the dimension 
of a linear space V and Im M, Ker M the image and the kernel of a matrix M, respectively. 


Theorem 14.5: 


There exists a unique symmetric solution X+. of Equation 14.15 such that all eigenvalues of A— BX + have 
nonpositive real part if and only if (A, B) is stabilizable. 


Riccati Equations and Their Solution 14-9 


Theorem 14.6: 


There exists a unique symmetric solution X_ of Equation 14.15 such that all eigenvalues of A— BX— have 
nonnegative real part if and only if (—A, B) is stabilizable. 


We observe that both (A, B) and (—A, B) are stabilizable if and only if (A, B) is controllable. It follows 
that both solutions X, and X_ exist if and only if (A, B) is controllable. 

For two real symmetric matrices X; and X2, the notation X; > Xz means that X; — Xz is nonnegative 
definite. Since A — BX has no eigenvalues with positive real part, neither has X; — X. Hence, X; — X > 
0. Similarly, one can show that X — X_ > 0, thus introducing a partial order among the set of symmetric 
solutions of Equation 14.15. 


Theorem 14.7: 


Suppose that X4 and X_ exist. If X is any symmetric solution of Equation 14.15, then 


Miro Se. 


That is why X and X_ are called the extreme solutions of Equation 14.15; X+ is the maximal symmetric 
solution, while X_ is the minimal symmetric solution. The set of all symmetric solutions of Equation 14.15 
can be related to a certain subset of the set of invariant subspaces of the matrix A — BX + or the matrix 
A — BX_. Denote Yo and V+ the invariant subspaces of A — BX that correspond, respectively, to the 
pure imaginary eigenvalues and to the eigenvalues having negative real part. Denote Wo and W_ the 
invariant subspaces of A — BX_ that correspond, respectively, to the pure imaginary eigenvalues and to 
the eigenvalues having positive real part. Then it can be shown that Vp = Wp is the kernel of X4 — X_ 
and the symmetric solution set corresponds to the set of all invariant subspaces of A — BX + contained in 
Vx or, equivalently, to the set of all invariant subspaces of A — BX_ contained in W_. 


Theorem 14.8: 


Suppose that X4 and X_ exist. Let X,, Xz be symmetric solutions of Equation 14.15 corresponding to the 
invariant subspaces V1, V2 of V+ (or Wi, W2 of W_). Then X, > X2 if and only if V; D V2 (or if and only 
ifWi Cc W)). 


This means that the symmetric solution set of Equation 14.15 is a complete lattice with respect to the 
usual ordering of symmetric matrices. The maximal solution X; corresponds to the invariant subspace 
V4 of A— BX or to the invariant subspace W = 0 of A—BX_, whereas the minimal solution X_ 
corresponds to the invariant subspace V = 0 of A — BX, or to the invariant subspace W_ of A — BX_. 

This result allows one to count the distinct symmetric solutions of Equation 14.15 in some cases. Thus, 
let a be the number of distinct eigenvalues of A — BX+ having negative real part and let m), m2,..., Ma 
be the multiplicities of these eigenvalues. Owing to the symmetries in H, the matrix A — BX_ exhibits the 
same structure of eigenvalues with positive real part. 


14-10 Control System Advanced Methods 


Theorem 14.9: 


Suppose that X+ and X_ exist. Then the symmetric solution set of Equation 14.15 has finite cardinality if 
and only if A — BX is cyclic on V (or if and only if A — BX_ is cyclic on W_.). In this case, the set contains 
exactly (m, +1)... (mg + 1) solutions. 


Simple examples are most illustrative. Consider Equation 14.15 with 


4=[0 tb a=[o al e=[o | 


and determine the lattice of symmetric solutions. We have 
XH (s) = st —55* +4 


and the following eigenvectors of H: 


wn or © 


are associated with the eigenvalues 1, —1, 2, and —2, respectively. Hence, the pair of solutions 
1 0 —-l1 0 
mele a} e=[o 3] 
Xu (s) = (s* — 3s + 2)(s* + 3s +2) 


1 0 —-1 0 
2 =[) =} xa=[5 | 


XH(s) = (s* —s— 2)(s? +5 —2). 


corresponds to the factorization 


and the solutions 


correspond to the factorization 


There are four subspaces invariant under the matrices 


A-x,=[ 7) a A-sx.=| 5 | 


each corresponding to one of the four solutions above. The partial ordering 
X,>X13>X-, Xp >Xi4 > X- 


defines the lattice visualized in Figure 14.2. 
As another example, we consider Equation 14.15 where 


a=[o o). 5=[h {). c=[h 9] 


and classify the symmetric solution set. 
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FIGURE 14.2 Lattice of solutions. 


We have 
XH(s) = (s— 1)?(s +1) 


and a choice of eigenvectors corresponding to the eigenvalues 1, —1 of H is 


>) 


Hence, 


are the extreme solutions. 
We calculate 


-1 0 1 0 
Asx. =| Al A-sx.=| 7 


and observe that the set of subspaces invariant under A — BX, or A — BX_ (other than the zero and the 
whole space, which correspond to X; and X_) is the family of one-dimensional subspaces parameterized 
by their azimuth angle 0. These correspond to the solutions 


eas cos 3 sin 
| sind = —cos 0’ 


Therefore, the solution set consists of X,, X_ and the continuous family of solutions X9. It is a complete 
lattice and X4 > X9 > X_ for every 9. 


14.4.3 Definite Solutions 


Under the standing assumption (Equation 14.12), namely 
B=B, B>0 and C=C’, C>0, 


one can prove that X; > 0 and X_ <0. The existence of X;, however, excludes the existence of X_ and 
vice versa, unless (A, B) is controllable. 

If X+ does exist, any other solution X > 0 of Equation 14.15 corresponds to a subspace W of W_ that 
is invariant under A — BX. From Equation 14.15, 


X(A= BX) + (AS BX) X = —=XBX = C, 


The restriction of A— BX to W has eigenvalues with positive real part. Since —XBX — C < 0, it follows 
from the Lyapunov theory that X restricted to W is nonpositive definite and hence zero. We conclude 
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that the solutions X > 0 of Equation 14.15 correspond to those subspaces W of WW_ that are invariant 
under A and contained in Ker C. 

The set of symmetric nonnegative definite solutions of Equation 14.15 is a sublattice of the lattice of all 
symmetric solutions [9,10]. Clearly X__ is the largest solution and it corresponds to the invariant subspace 
W = 0 of A. The smallest nonnegative definite solution will be denoted by X,, and it corresponds to W,., 
the largest invariant subspace of A contained in Ker C and associated with eigenvalues having positive 
real part. 

The nonnegative definite solution set of Equation 14.15 has finite cardinality if and only if A is cyclic 
on W,,. In this case, the set contains exactly (pj + 1)... ( +1) solutions, where p is the number of distinct 
eigenvalues of A associated with W,, and pj, p2,..., Pp are the multiplicities of these eigenvalues. 

Analogous results hold for the set of symmetric solutions of Equation 14.15 that are nonpositive 
definite. In particular, if X_ exists, then any other solution X < 0 of Equation 14.15 corresponds to a 
subspace V of V+ that is invariant under A and contained in Ker C. Clearly X_ is the smallest solution 
and it corresponds to the invariant subspace V = 0 of A. The largest nonpositive definite solution is 
denoted by Xx and it corresponds to Wx, the largest invariant subspace of A contained in Ker C and 
associated with eigenvalues having negative real part. 

Let us illustrate this with a simple example. Consider Equation 14.15 where 


t=[o i} P=[o a} e=[o ol 


and classify the two sign-definite solution sets. We have 


Thus, there are three invariant subspaces of W,. corresponding to the three nonnegative definite 
solutions of Equation 14.15 


These solutions make a lattice and 


Kee ee x 


The matrix A has no eigenvalues with negative real part. Therefore, V,, = 0 and X_ is the only nonpos- 
itive definite solution of Equation 14.15. 
Another example for Equation 14.15 is provided by 


s=[b a): B=[o of C=[o ol: 
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It is seen that neither (A, B) nor (—A, B) is stabilizable; hence, neither X, nor X_ exists. The symmetric 
solution set consists of one continuous family of solutions 


for any real a. Therefore, both sign-definite solution sets are infinite; the nonnegative solution set is 
unbounded from above while the nonpositive solution set is unbounded from below. 


14.5 Limiting Behavior of Solutions 


The length of the time interval t) — tf; in the optimal control and filtering problems is rather artificial. 
For this reason, an infinite time interval is often considered. This brings in the question of the limiting 
behavior of the solution X(t, t2, T) for the Riccati differential equation 14.11. 

In applications to optimal control, it is customary to fix t and let f2 approach +00. Since the coefficient 
matrices of Equation 14.11 are constant, the same result is obtained if tz is held fixed and t approaches 
—oo. The limiting behavior of X(t, t2, T) strongly depends on the terminal matrix T > 0. For a suitable 
choice of T, the solution of Equation 14.11 may converge to a constant matrix X > 0, a solution of 
Equation 14.15. For some matrices T, however, the solution of Equation 14.11 may fail to converge to a 
constant matrix, but it may converge to a periodic matrix function. 


Theorem 14.10: 


Let (A, B) be stabilizable. Ift and T are held fixed and tz > 00, then the solution X(t, tz, T) of Equation 14.11 
is bounded on the interval [t, oo). 


This result can be proved by associating an optimal control problem with Equation 14.11. Then stabi- 
lizability of (A, B) implies the existence of a stabilizing (not necessarily optimal) control. The consequent 
cost functional of Equation 14.4 is finite and dominates the optimal one. 

If (A, B) is stabilizable, then X, exists and each real symmetric nonnegative definite solution X of 
Equation 14.15 corresponds to a subset W of W.,,, the set of A-invariant subspaces contained in Ker C 
and associated with eigenvalues having positive real part. The convergence of the solution X(t, tz, T) of 
Equation 14.11 to X depends on the properties of the image of W,, under T, see [11]. 

For simplicity, it is assumed that the eigenvalues 1, d2,..., Xp» of A associated with W,. are simple 
and, except for pairs of complex conjugate eigenvalues, have different real parts. Let the corresponding 
eigenvectors be ordered according to decreasing real parts of the eigenvalue 


V1,V2>-- +» Vp» 


and denote WV; the A-invariant subspace of W, spanned by v1, v2,..., Vj. 


Theorem 14.11: 


Let (A, B) be stabilizable and the subspaces Wj of W, satisfy the above assumptions. Then, for all fixed t 
and a given terminal condition T > 0, the solution X(t, t2, T) of Equation 14.11 converges to a constant 
solution of Equation 14.15 as tz > 00 if and only if the subspace W;,.4., corresponding to any pair hp, he+1 
of complex conjugate eigenvalues is such that dim TW,.4. equals either dim TW,_, or dim TWy_1 +2. 
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Here is a simple example. Let 


eee ee eee) 


The pair (A, B) is stabilizable and A has two eigenvalues 1 +j and 1 —j. The corresponding eigenvectors 


mf =f 


span W,,.. Now consider the terminal condition 


Then, 


1 
TWo = 0, TW) =Im B ‘ 
Theorem 14.11 shows that X(t, t2, T) does not converge to a constant matrix; in fact, 


1 eee tr) ce | 


X(t, t2, T) = ——~—~ A E 
(t,t2, T) 1+ e%t-) |—sin2(t—t)) 2 sin?(t — t) 


tends to a periodic solution if tj + oo. On the other hand, if we select 
0 0 
wl ol 


ToWo = 0, ToW2 = 0 


we have 


and X(t, tz, To) does converge. Also, if we take 


we have 
TiWo =0, T1W2= R? 


and X(t, tz, T) converges as well. 

If the solution X(t, t2, T) of Equation 14.11 converges to a constant matrix X7 as tz > oo, then Xr 
is a real symmetric nonnegative definite solution of Equation 14.15. Which solution is attained for a 
particular terminal condition? 


Theorem 14.12: 


Let (A, B) be stabilizable. Let 
Xr = lim X(t, th, T) 
tl co 


for a fixed T >0. Then X7 >0 is the solution of Equation 14.15 corresponding to the subspace Wr of W,, 
defined as the span of the real vectors v; such that TW; = TW,-_, and of the complex conjugate pairs vx, 
Vp41 Such that TWh.) = TWe-1. 
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The cases of special interest are the extreme solutions X; and X,. The solution X(t, t, T) of Equa- 
tion 14.12 tends to X+ if and only if the intersection of W,, with Ker T is zero, and to X,. if and only if W,. 
is contained in Ker T. 

This is best illustrated in the previous example, where 


Bele 4 eBay ape eelncul 


and W, = R?. Then X(t, t2, T) converges to X+ if and only if T is positive definite; for instance, the 
identity matrix T yields the solution 


iS —* F He 


1 + e2(t-h) 0 1 
which tends to 
2 0 
oe F | 
On the other hand, X(t, 2, T) converges to X, if and only if T=0; then 
X(t, t2,0) =0 


and X,.=0 is a fixed point of Equation 14.11. 


14.6 Optimal Control and Filtering: Application 


The problems of optimal control and filtering introduced in Section 14.2 are related to the matrix Riccati 
differential equations 14.5 and 14.9, respectively. These problems are defined over a finite horizon tz — ty. 
We now apply the convergence properties of the solutions to study the two optimal problems in case the 
horizon becomes large. 

To fix ideas, we concentrate on the optimal control problem. The results can easily be interpreted in 
the filtering context owing to the duality between Equations 14.5 and 14.9. 

We recall that the finite horizon optimal control problem is that of minimizing the cost functional of 
Equation 14.4, 


(br) = [vio + u'(t)u(t)] dt + x'(t,)Tx(t2) 

along the solutions of Equation 143, 

x(t) = Fx(t) + Gu(t) 

y(t) = Hx(t). 

The optimal control has the form 
ug(t) = —G’X(t, ta, T)x(t), 
where X(t, tz, T) is the solution of Equation 14.11, 
X(t) + X(DA+A'X(t) — X(HBX(t) + C= 0 


subject to the terminal condition X(t2) = T, and where 
A=F, B=GG, C=H'H. 
The optimal control can be implemented as a state feedback and the resulting closed-loop system is 
x(t) = [F — GG'X(t, to, T)]x(t) 
= [A— BX(t, to, T)]x(t). 


Hence, the relevance of the matrix A — BX, which plays a key role in the theory of the Riccati equation. 
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The infinite horizon optimal control problem then amounts to finding 


Nx =inf lim 7(t2) (14.17) 


u(t) 200 


and the corresponding optimal control u,,.(t), f > t; achieving this minimum cost. 
The receding horizon optimal control problem is that of finding 


Nx = jim inf (tz) (14.18) 
2 


> u(t) 


and the limiting behavior u,,,(t), t > t, of the optimal control u(t). 

The question is whether 7, is equal to 1,.,.and whether u,. coincides with u,.,.. If so, the optimal control 
for the infinite horizon can be approximated by the optimal control of the finite horizon problem for a 
sufficiently large time interval. 

It turns out [12] that these two control problems have different solutions corresponding to different 
solutions of the matrix Riccati algebraic Equation 14.15, 


XA+A’X —XBX+C=0. 


Theorem 14.13: 
Let (A, B) be stabilizable. Then the infinite horizon optimal control problem of Equation 14.17 has a solution 
Ya = x'(t1)Xox(t1), usx(t) = —G'Xox(t) 


where X, >0 is the solution of Equation 14.15 corresponding to Wo, the largest A-invariant subspace 
contained in W,,N Ker T. 


Theorem 14.14: 


Let (A, B) be stabilizable. Then the receding horizon optimal control problem of Equation 14.18 has a 
solution if and only if the criterion of Theorem 14.5 is satisfied and, in this case, 


Nx =X'(t))Xrx(t1),  Uxx(t) = —G'X x(t) 
where Xr > 0 is the solution of Equation 14.16 corresponding to Wr and defined in Theorem 14.5. 


The equivalence result follows. 


Theorem 14.15: 


The solution of the infinite horizon optimal control problem is exactly the limiting case of the receding 
horizon optimal control problem if and only if the subspace W,.M KerT is invariant under A. 
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A simple example illustrates these points. Consider the finite horizon problem defined by 


X1(t) = 2x; (t) + u(t), 
Xo(t) = x2(t) + u2(t) 


and 


S 
io =A eetores: i. [u2(t) + w2(n)] de 


to 


which corresponds to the data 


and 


Clearly W, = R* and the subspace 


W,. 1 Ker T = Im | 


is not invariant under A. Hence, the infinite and receding horizon problems are not equivalent. 
The lattice of symmetric nonnegative definite solutions of Equation 14.11 has the four elements 


4 0 0 0 4 0 0 0 
slo a [oa] o=[o a> [0 a] 


depicted in Figure 14.3. 
Since the largest A-invariant subspace of W,, KerT is zero, the optimal solution X, of Equation 14.11 
is the maximal element X;. The infinite horizon optimal control reads 


Uj(t) = —4x1(t), 
Uz (t) = —2x2(t), 


and affords the minimum cost 
Nx = 4x7 (th) + 2x3(t). 


Now the eigenvectors of A spanning WV, are 


Ek Ll 


FIGURE 14.3 The four elements of the lattice of solutions. 
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mel) ml 


are linearly dependent. Hence, Wr is spanned by v2 only, 


Wr =Im "| 


and the optimal limiting solution X7 of Equation 14.11 equals X2. The receding horizon optimal control 
reads 


and their T-images 


Ujxx(t) = —4x) (t) 
U2xx(t) = 0 


and affords the minimum cost 


Nax(t) = 4x? (th). 


The optimal control problems with large horizon are practically relevant if the optimal closed-loop 
system 
x(t) = (A — BX)x(t) 


is stable. A real symmetric nonnegative definite solution X of Equation 14.15 is said to be stabilizing if 
the eigenvalues of A — BX all have negative real part. It is clear that the stabilizing solution, if it exists, 
is the maximal solution X,. Thus, the existence of a stabilizing solution depends on A — BX_. having 
eigenvalues with only negative real part. 


Theorem 14.16: 


Equation 14.15 has a stabilizing solution if and only if (A, B) is stabilizable and the Hamiltonian matrix H 
of Equation 14.16 has no pure imaginary eigenvalue. 


The optimal controls over large horizons have a certain stabilizing effect. Indeed, if X >0 is a solution of 
Equation 14.15 that corresponds to an A-invariant subspace W of W,, then the control u(t) = —G’Xx(t) 
leaves unstable in A — BX just the eigenvalues of A associated with W; all the remaining eigenvalues of 
A with positive real part are stabilized. Of course, the pure imaginary eigenvalues of A, if any, cannot be 
stabilized; they remain intact in A — BX for any solution X of Equation 14.15. 

In particular, the infinite horizon optimal control problem leaves unstable the eigenvalues of A asso- 
ciated with W,, which are those not detectable either in C or in T, plus the pure imaginary eigenvalues. 
It follows that the infinite horizon optimal control results in a stable system if and only if X, is the stabi- 
lizing solution of Equation 14.15. This is the case if and only if the hypotheses of Theorem 14.6 hold and 
Wo, the largest A-invariant subspace contained in W,M KerT, is zero. Equivalently, this corresponds to 
the pair 


being detectable. 

The allocation of the closed-loop eigenvalues for the receding horizon optimal control problem is 
different, however. This control leaves unstable all eigenvalues of A associated with Wry, where Wr is 
a subspace of W, defined in Theorem 14.5. Therefore, the number of stabilized eigenvalues may be 
lower, equal to the dimension of TW, whenever Ker T is not invariant under A. It follows that the 
receding horizon optimal control results in a stable system if and only if Xr is the stabilizing solution 
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of Equation 14.15. This is the case if and only if the hypotheses of Theorem 14.6 hold and Wy is zero. 
Equivalently, this corresponds to W,,M KerT = 0. Note that this case occurs in particular if T > X_. 
It further follows that under the standard assumption, namely that 


(A, B) stabilizable 
(A, C) detectable, 


both infinite and receding horizon control problems have solutions; these solutions are equivalent for 
any terminal condition T; and the resulting optimal system is stable. 


14.7 Numerical Solution 


The matrix Riccati differential equation 14.11 admits an analytic solution only in rare cases. A numerical 
integration is needed and the Runge-Kutta methods can be applied. 

A number of techniques are available for the solution of the matrix Riccati algebraic Equation 14.15. 
These include invariant subspace methods [13] and the matrix sign function iteration [14]. We briefly 
outline these methods here with an eye on the calculation of the stabilizing solution to Equation 14.15. 


14.7.1 Invariant Subspace Method 


In view of Theorem 14.4, any solution X of Equation 14.15 can be computed from a Jordan form 
reduction of the associated 2n x 2n Hamiltonian matrix 


A -—B 
ele | 
Specifically, compute a matrix of eigenvectors V to perform the following reduction: 
1 _|-J 0 
V-HAV= | 0 i ; 


where —J is composed of Jordan blocks corresponding to eigenvalues with negative real part only. If 
the stabilizing solution X exists, then H has no eigenvalues on the imaginary axis and J is indeed n x n. 


Writing 
Vin a 
V= , 
i V2 
where each Vj is n x n, the solution sought is found by solving a system of linear equations, 
X=Vn Vj). 


However, there are numerical difficulties with this approach when H has multiple or near-multiple 
eigenvalues. To ameliorate these difficulties, a method has been proposed in which a nonsingular matrix 
V of eigenvectors is replaced by an orthogonal matrix U of Schur vectors so that 


Si | 


’HU = 
U'HU E = 


where now Sj, is a quasi-upper triangular matrix with eigenvalues having negative real part and Sy is a 
quasi-upper triangular matrix with eigenvalues having positive real part. When 


Ui, Uy 
U — 5 
B ma 
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he 
Var |> | Ua1 


span the same invariant subspace and X can again be computed from 


we observe that 


XS Unis 


14.7.2 Matrix Sign Function Iteration 


Let M be a real n x n matrix with no pure imaginary eigenvalues. Let M have a Jordan decomposition 
M=VJV~'andlet dy, X2,...5 dy be the diagonal entries of J (the eigenvalues of M repeated according 
to their multiplicities). Then the matrix sign function of M is given by 


sgn Red, 
senM=V ae yo? 
sgn Rehn 


It follows that the matrix Z = sgn M is diagonalizable with eigenvalues +1 and Z” = I. The key 
observation is that the image of Z + I is the M-invariant subspace of R” corresponding to the eigenvalues 
of M with negative real part. 

This property clearly provides the link to Riccati equations, and what we need is a reliable computation 
of the matrix sign. Let Zj = M be ann x n matrix whose sign is desired. For k = 0, 1, perform the iteration 


1 
Zp = —(Z,+0°Z, 1), 
k+1 5! k k ) 


where c = |det Z,|1/. Then 
lim Z, =Z=sgnM. 
k->0o 
The constant c is chosen to enhance convergence of this iterative process. If c = 1, the iteration amounts 
to Newton’s method for solving the equation 


Z*-I=0. 


Naturally, it can be shown that the iteration is ultimately quadratically convergent. 

Thus, to obtain the stabilizing solution X of Equation 14.15, provided it exists, we compute Z = sgn 
H, where H is the Hamiltonian matrix of Equation 14.16. The existence of X guarantees that H has no 
eigenvalues on the imaginary axis. 

Writing 


Z\1 zs 
L= ; 
B 222 


where each Z; is n x n, the solution sought is found by solving a system of linear equations 


Z12 —  {4Zu+1 
aetil*=- [an | 
14.7.3 Concluding Remarks 


We have discussed two numerical methods for obtaining the stabilizing solution of the matrix Riccati 
algebraic equation 14.15. They are both based on the intimate connection between the Riccati equation 
solutions and invariant subspaces of the associated Hamiltonian matrix. The method based on Schur 
vectors is a direct one, while the method based on the matrix sign function is iterative. 
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The Schur method is now considered one of the more reliable for Riccati equations and has the virtues 
of being simultaneously efficient and numerically robust. It is particularly suitable for Riccati equations 
with relatively small dense coefficient matrices, say, of the order of a few hundreds or less. The matrix 
sign function method is based on the Newton iteration and features global convergence, with ultimately 
quadratic order. Iteration formulas can be chosen to be of arbitrary order convergence in exchange for, 
naturally, an increased computational burden. The effect of this increased computation can, however, be 
ameliorated by parallelization. 

The two methods are not limited to computing the stabilizing solution only. The matrix sign iteration 
can also be used to calculate X_, the antistabilizing solution of Equation 14.15, by considering the 
matrix sgn H — I instead of sgn H + I. The Schur approach can be used to calculate any, not necessarily 
symmetric, solution of Equation 14.15, by ordering the eigenvalues on the diagonal of S accordingly. 
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15.1 Introduction 


An observer for a dynamic system S(x,y, u) with state x, output y, and input u is another dynamic system 
S(&,y, u) having the property that the state % of the observer S converges to the state x of the process S, 
independent of the input u or the state x. 

Among the various applications for observers, perhaps the most important is for the implementation of 
closed-loop control algorithms designed by state-space methods. The control algorithm is designed in two 
parts: a “full-state feedback” part based on the assumption that all the state variables can be measured; and 
an observer to estimate the state of the process based on the observed output. The concept of separating 
the feedback control design into these two parts is known as the separation principle, which has rigorous 
validity in linear systems and in a limited class of nonlinear systems. Even when its validity cannot be 
rigorously established, the separation principle is often a practical solution to many design problems. 

The concept of an observer for a dynamic process was introduced in 1966 by Luenberger [1]. The 
generic “Luenberger observer,” however, appeared several years after the Kalman filter, which is in fact 
an important special case of a Luenberger observer—an observer optimized for the noise present in the 
observations and in the input to the process. 


15.2 Linear Full-Order Observers 


15.2.1 Continuous-Time Systems 


Consider a linear, continuous-time dynamic system 


x= Ax+ Bu (15.1) 
y=Cx (15.2) 


15-1 
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The more generic output 
y=Cx+ Du 


can be treated by defining a modified output 
y=y—Du 
and working with y instead of y. The direct coupling Du from the input to the output is absent in most 


physical plants. 
A full-order observer for the linear process defined by Equations 15.1 and 15.2 has the generic form 


X= AR+Ky+Hu (15.3) 


where the dimension of state X of the observer is equal to the dimension of process state x. 

The matrices A, K, and H appearing in Equation 15.3 must be chosen to conform with the required 
property of an observer: that the observer state must converge to the process state independent of the 
state x and the input u. To determine these matrices, let 


e:=x—-X (15.4) 


be the estimation error. From Equations 15.1 through 15.3 


é@=Ax+ Bu— A(x — e) — GCx — Hu, 
= Ae+(—-A+A—KC)x+(B—H)u, (15.5) 


From Equation 15.5 it is seen that for the error to converge to zero, independent of x and u, the 
following conditions must be satisfied: 


A=A-—KC (15.6) 
H=B (15.7) 


When these conditions are satisfied, the estimation error is governed by 
é—Ae (15.8) 


which converges to zero if A is a “stability matrix.” When A is constant, this means that its eigenvalues 
must lie in the (open) left half-plane. 

Since the matrices A, B, and C are defined by the plant, the only freedom in the design of the observer 
is in the selection of the gain matrix K. 

To emphasize the role of the observer gain matrix, and accounting for requirements of Equations 15.6 
and 15.7, the observer can be written as 


& = AX + Bu+ K(y— C2) (15.9) 


A block diagram representation of Equation 15.9, as given in Figure 15.1, aids in the interpretation of 
the observer. Note that the observer comprises a model of the process with an added input: 


K(y — Cx) = Kr 
The quantity 

ri=y—Cx=y-y (15.10) 
often called the residual, is the difference between the actual observation y and the “synthesized” obser- 
vation 

y = CK 

produced by the observer. The observer can be viewed as a feedback system designed to drive the residual 
to zero: as the residual is driven to zero, the input to Equation 15.9 due to the residual vanishes and the 
state of Equation 15.9 looks like the state of the original process. 
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u | Control input 


ae E: 
Observation A 


Plant model 


FIGURE 15.1 Full-order observer for linear process. 


The fundamental problem in the design of an observer is the determination of the observer gain matrix 
K such that the closed-loop observer matrix 


A=A-—KC (15.11) 


is a stability matrix. 
There is considerable flexibility in the selection of the observer gain matrix. Two methods are standard: 
optimization and pole placement. 


15.2.1.1 Optimization 


Since the observer given by Equation 15.9 has the structure of a Kalman filter (see Chapter 13), its gain 
matrix can be chosen as a Kalman filter gain matrix, that is, 


K=PC'R (15.12) 
where P is the covariance matrix of the estimation error and satisfies the matrix Riccati equation 
P= AP + PA’ — PC’R"'CP+Q, (15.13) 


where R is a positive-definite matrix and Q is a positive-semidefinite matrix. 

In most applications the steady-state covariance matrix is used in Equation 15.12. This matrix is 
given by setting P in Equation 15.13 to zero. The resulting equation is known as the algebraic Riccati 
equation. Algorithms to solve the algebraic Riccati equation are included in popular control system 
software packages such as MATLAB® or SciLab. 

In order for the gain matrix given by Equations 15.12 and 15.13 to be genuinely optimum, the process 
noise and the observation noise must be white with the matrices Q and R being their spectral densities. 
It is rarely possible to determine these spectral density matrices in practical applications. Hence, the 
matrices Q and R are best treated as design parameters that can be varied to achieve overall system design 
objectives. 
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If the observer is to be used as a state estimator in a closed-loop control system, an appropriate form 
for the matrix Q is 
Q=4q' BB (15.14) 


As has been shown by Doyle and Stein [2], as q — 00, this observer tends to “recover” the stability 
margins assured by a full-state feedback control law obtained by quadratic optimization. 


15.2.1.2 Pole Placement 


An alternative to solving the algebraic Riccati equation to obtain the observer gain matrix is to select K 
to place the poles of the observer, that is, the eigenvalues of A in Equation 15.11. 

When there is a single observation, K is a column vector with exactly as many elements as eigenvalues 
of A. Hence, specification of the eigenvalues of A uniquely determines the gain matrix K. A number of 
algorithms can be used to determine the gain matrix, some of which are incorporated into the popular 
control system design software packages. Some of the algorithms have been found to be numerically 
ill-conditioned; so caution should be exercised in using the results. 

The present author has found the Bass—Gura [3] formula to be effective in most applications. This 
formula gives the gain matrix as 


K =(OW)’!(4—a) (15.15) 
where 


a=[a a, +: an]. (15.16) 
is the vector formed from the coefficients of the characteristic polynomial of the process matrix A 
|s1 —A| = s" + ays"! +---+ay_istay (15.17) 
and @ is the vector formed from the coefficients of the desired characteristic polynomial 
|s] — A] =s" +s" +--+ Gy_is +p (15.18) 
The other matrices in Equation 15.15 are given by 
O= [CO AIC’ ase Are] (15.19) 


which is the observability matrix of the process, and 


1 a an 
0 1 st Ay] 

W=]... : : (15.20) 
0 0 1 


The determinant of W is 1; hence it is not singular. If the observability matrix O is not singular, the 
inverse matrix required in Equation 15.15 exists. Hence, the gain matrix K can be found that places 
the observer poles at arbitrary locations if (and only if) the process for which an observer is sought 
is observable. Numerical problems occur, however, when the observability matrix is nearly singular. 
Other numerical problems can arise in determination of the characteristic polynomial |sJ — A| for high- 
order systems and in the determination of s! — A when the individual poles, and not the characteristic 
polynomial, are specified. In such instances, it may be necessary to use an algorithm designed to handle 
difficult numerical calculations. 

When two or more quantities are observed, there are more elements in the gain matrix than eigenvalues 
of A; hence specification of the eigenvalues of A does not uniquely specify the gain matrix K. In addition 
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to placing the eigenvalues, more of the “eigenstructure” of A can be specified. This method of selecting 
the gain matrix is fraught with difficulty, however, and the use of the algebraic Riccati equation is usually 
preferable. 

The Matlab algorithm place can be used to place the poles of a system with multiple observations, and 
uses the additional freedom to improve the robustness of the resulting observer. 


15.2.2 Delayed Data 
In some applications, the observation data may be delayed: 
ya(t)=y(t—T), with y(t) = Cx(t) (15.21) 


in which T is the known time delay. 

If the observation vector y(t) has more than one component, then it is assumed that all the components 
are delayed by the same amount of time T. 

On recognizing that the process is time-invariant, that is, 


x(t — T) = Ax(t — T)+ Bu(t — T) (15.22) 


an observer for the delayed state is given by 
X(t — T) = AX(t — T) + Bu(t — T) + K(yq(t) — CX(t — T)) (15.23) 


where K is the observer gain matrix, which can be obtained by the methods discussed above. 

Having thus obtained an estimate x(t — T) of the delayed state, we extrapolate the estimate to the 
present time. Since there are no data in the interval [t — T, t], the extrapolation simply makes use of the 
state transition matrix e4¢", with 

A, =A-—BG 
of the closed-loop system over the interval [t — T, T]. Thus the estimate of the state at time f is given by 
R(t) = eT R(t — T) (15.24) 


where x(t) is given by Equation 15.23. 
The observer requires the delayed control signal u(t — T), which can be implemented by any of the 
“standard” methods for implementing a delay line. 


15.3 Linear Reduced-Order Observers 


The observer described in the previous section has the same order as the plant, irrespective of the number 
of independent observations. A reduced-order observer of the order n — m, where nis the dimension of the 
state vector and m is the number of observations, can also be specified. When the number of observations is 
comparable to the dimension of the state vector, the reduced-order observer may represent a considerable 
simplification. 

The description of the reduced-order observer is simplified if the state vector can be partitioned into 
two substates: 


gear iso (15.25) 


such that 
xj =y=Cx (15.26) 


is the observation vector (of dimension m) and x2 (of dimension n — m) comprises the components of 
the state vector that cannot be measured directly. 
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In terms of x), and x2, the plant dynamics are written as 


Xy = Ay x] + Ayox2 + Byu 


X2 = A21x1 + A22%2 + Bou 
Since x; is directly measured, no observer is required for that substate, that is, 
Xp =x =y 
For the remaining substate, we define the reduced-order observer by 
X= Ky+z 
where z is the state of a system of order n — m: 
z=Az+Ly+Hu 


A block-diagram representation of the reduced-order observer is given in Figure 15.2a. 


(a) yam a, 


> 
Observation 
u x Zz ey 
Mt O— Oa 
Control K 


input 
A I< 
(b) yar, aT 
@ > 
Observation 
Y 
L K 


sf}-4 p+ 
Control K 


input 


FIGURE 15.2 Reduced-order observer for linear process. (a) Feedback from z and (b) feedback from x2. 


(15.27) 
(15.28) 


(15.29) 


(15.30) 


(15.31) 
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The matrices A, L,H, and K are chosen, as in the case of the full-order observer, to ensure that the 
error in the estimation of the state converges to zero, independent of x, y, and u. 
Since there is no error in the estimation of x1, that is, 


e] =x} —x, =0 (15.32) 
by virtue of Equation 15.29, it is necessary to ensure only the convergence of 
€2 = X2 — X2 (15.33) 


to zero. 
From Equations 15.28 through 15.31 


€) = (Az; — KA, +AK L)x, + (Ax — KAj2 A)x + Ae, + (B2 — KB, — H)u (15.34) 


As in the case of the full-order observer, to make the coefficients of x), x2, and u vanish it is necessary 
that the matrices in Equations 15.29 and 15.31 satisfy 


a 


A= Ay — KA), (15.35) 
L= A) — KA, + AK, (15.36) 
H = By — KB, (15.37) 


Two of these conditions (Equations 15.35 and 15.37) are analogous to Equations 15.6 and 15.7 for the 
full-order observer; Equation 15.36 is a new requirement for the additional matrix L that is required by 
the reduced-order observer. 

When these conditions are satisfied, the error in estimation of x2 is given by 


A 


e2 = Ae. 


Hence, the gain matrix K must be chosen such that the eigenvalues of A= Az2 — KAjz lie in the (open) 
left half-plane; A22 and Aj2 in the reduced-order observer take the roles of A and C in the full-order 
observer; once the gain matrix K is chosen, then there is no further freedom in the choice of L and H. 

The specific form of the new matrix L in Equation 15.36 suggests another option for implementation 
of the dynamics of the reduced-order observer, namely 


z= Ax +Ly+ Hu, (15.38) 


where 


LE =A, —KAy (15.39) 


A block-diagram representation of this option is given in Figure 15.2b. 

The selection of the gain matrix K of the reduced-order observer may be accomplished by any of the 
methods that can be used to select the gains of the full-order observer. In particular, pole placement, using 
any convenient algorithm, is feasible. Or the gain matrix can be obtained as the solution of a reduced- 
order Kalman filtering problem. For this purpose, one can use Equations 15.12 and 15.13, with A and C 
therein replaced by Azz and Aj of the reduced-order problem. 
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A more rigorous solution, taking into account the cross-correlation between the observation noise and 
the process noise [4], is available. Suppose the dynamic process is governed by 


x) = Aq x) + Ajox2 + Byut Fv, (15.40) 
x2 = Az) x) + A22x2 + Bout Foy, (15.41) 
with the observation being noise free: 
y=x) (15.42) 
In this case, the gain matrix is given by 
K = (PA), + F,QF))R, (15.43) 
where 
R=F\QF, 


and P is the covariance matrix of the estimation error e2, as given by 


P=AP+PA'—PAyR'A),P+Q, (15.44) 

where 
A= An — F.QF,{R At, (15.45) 
Q = FyQF) — FxQF,R1F\ QF; (15.46) 


Note that Equation 15.44 becomes homogeneous when 


Q=0 (15.47) 
One of the solutions of Equation 15.44 could be 
P=0. (15.48) 


which would imply that the error in estimating x2 converges to zero! We cannot expect to achieve anything 
better than this. Unfortunately, P = 0 is not the only possible solution to Equation 15.47. To test whether 
it is, it is necessary to check whether the resulting observer dynamics matrix 


A= Ay — FoF, 'Ay (15.49) 


is a stability matrix. If not, Equation 15.47 is not the correct solution to Equation 15.44. 

The eigenvalues of the “zero steady-state variance” observer dynamics matrix Equation 15.49 have an 
interesting interpretation: as shown in [4], these eigenvalues are the transmission zeros of the plant with 
respect to the noise input to the process. Hence, the variance of the estimation error converges to zero if 
the plant is “minimum phase” with respect to the noise input. 

For purposes of robustness, as discussed in Section 4, suggest that the noise distribution matrix F 
includes a term proportional to the control distribution matrix B, that is, 


F=F+q°BB 
In this case, the zero-variance observer gain would satisfy 
H = B, — KB, =0, (15.50) 


as q —> 00. 
If Equation 15.50 is satisfied, the observer poles are located at the transmission zeros of the plant. Thus, 

in order to use the gain given by Equation 15.50, it is necessary for the plant to be minimum phase with 

respect to the input. Rynaski [5] has defined observers meeting this requirement as robust observers. 
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15.4 Discrete-Time Systems 


Observers for discrete-time systems can be defined in a manner analogous to continuous-time systems. 
Consider a discrete-time linear system 


Xn-1 = Ox, +Tuy (15.51) 


with observations defined by 
Yn = CXp (15.52) 


A full-order observer for Equation 15.51 is a dynamic system of the same order as the process whose 
state is to be estimated, excited by the inputs and outputs of that process and having the property that the 
estimation error (i.e., the difference between the state x, of the process and the state x, of the observer) 
converges to zero as 1 —> 00, independent of the state of the process or its inputs and outputs. 

Let the observer be defined by the general linear difference equation 


Kno = Bq + Kyy + Huy (15.53) 


The goal is to find conditions on the matrices ®, K, and H such that the requirements stated above are 
met. To find these conditions subtract Equation 15.53 from Equation 15.51 


Xn+1 —Xn+1 = Oxy, + Tuy o%,, Ky, — Huy, (15.54) 


Letting 
en = Xn — Xn 


and using Equation 15.52 we obtain from Equation 15.54 


en41 = Pe, + (® — KC— &)x, + (TC — H) uy (15.55) 


Thus, in order to meet the requirements stated above, the transition matrix of the observer must be 
stable (ie., the eigenvalues of ® must lie within the unit circle) and, moreover, 


= ®-—KC, (15.56) 
=. (15.57) 


Ze 


By virtue of these relations the observer can be expressed as 


Xnt1 = Oxy + Tun + K(yn — CXn) (15.58) 


It is seen from Equation 15.58 that the observer has the same dynamics as the underlying process, 
except that it has an additional input 
K(yn — Cn), 


that is, a gain matrix K multiplying the residual 
tn = Vn — CXn 


As in the continuous-time case, the observer can be interpreted as a feedback system, the role of which 
is that of driving residual r, to zero. 

The observer design thus reduces to the selection of the gain matrix K that makes the eigenvalues of 
= © — KC lie at suitable locations within the unit circle. 

If the discrete-time system is observable, the eigenvalues of ®, = @ — KC can be put anywhere. For a 
single-output plant, the Bass—Gura formula or other well-known algorithm can be used. For both single 
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and multiple output processes, the observer gain matrix can be selected to make the observer a Kalman 
filter (i.e., a minimum variance estimator). 
The gain matrix of the discrete-time Kalman filter is given by 


K = ®PC'(PCP’ + R)~, (15.59) 


where P is the covariance matrix of the estimation error, given (in the steady state) by the discrete-time 
algebraic Riccati equation 


P = ®[P — PC'(CPC’ +R) 'CP]&’+Q (15.60) 


The matrices Q and R are the covariance matrices of the excitation noise and the observation noise, 
respectively. As in the case of continuous-time processes, it is rarely possible to determine these matrices 
with any degree of accuracy. Hence, these matrices can be regarded as design parameters that can be 
adjusted by the user to provide desirable observer characteristics. 


15.4.1 Delayed Data 

Consider the discrete-time system defined by Equation 15.51 and with delayed observations defined by 
Zn. = Cx; 
Yn = 2n—-N> 


where N is the time delay, assumed to be an integer. The method described in Section 15.2.2 can readily 
be extended to this case. As an alternative the following method can be used. Define the metastate as the 
v+kN dimensioned vector (where v is the dimension of the state vector and k is the dimension of the 
original observation vector) 


Xx. ms , 
X,= al with z, = [Zin Zon +e ztn| 


Then the metatstate evolves according to 


Xn¢1 = Ox, + Tuy, 


where 
® 0 O ::- 0 0 
C 0 0 0 
®=| 0 I 0 0 0 
0 0 0 I 0 
and 
PS [PO re 0) 07 


For all components of the observation vector concurrent, the observation equation is 
j= CSO Oe OTe 
For m time delays (each an integer multiple of the sampling time), the observation matrix is given by 
C=[0 whe Up Sie UR aks Um] » 


where Uj is a column vector with ones in the positions of the observation components present at the 
ith observation and zeros otherwise. Applying the theory developed earlier to the metasystem gives the 
observer the presence of delayed data. 

It is noted that implementation of this method requires the storage of past observations, but permits 
the treatment of multiple observations with different time delays. 
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15.5 The Separation Principle 


The predominant use of an observer is to estimate the state for purposes of feedback control. In particular, 
in a linear system with a control designed on the assumption of full-state feedback 


u=—Gx, (15.61) 


when the state x is not directly measured, the state x of the observer is used in place of the actual state x 
in Equation 15.61. Thus, the control is implemented using 


u=—G%, (15.62) 


where 
k=x-e (15.63) 


Hence, when an observer is used, the closed-loop dynamics are given in part by 


x = Ax — BG(x — e) = (A — BG)x + BGe (15.64) 


This equation, together with the equation for the propagation of the error, defines the complete 
dynamics of the closed-loop system. 
When a full-order observer is used 


@—Ae=(A—KC)e (15.65) 


Thus, the complete closed-loop dynamics are 


x A—BG BG x 
Le 0 
The closed-loop dynamics are governed by the upper triangular matrix 
A—BG BG 
a=[ : ae (15.67) 


the eigenvalues of which are given by 
|sf — A| = |s1 —A + BG||sI - A+KC|=0, (15.68) 


that is, the closed-loop eigenvalues are the eigenvalues of A — BG, the full-state feedback system; and 
the eigenvalues of A — KC, the dynamics matrix of the observer. This is a statement of the well-known 
separation principle, which permits one to design the observer and the full-state feedback control inde- 
pendently, with the assurance that the poles of the closed-loop dynamic system will be the poles selected 
for the full-state feedback system and those selected for the observer. 

When a reduced-order observer is used, it is readily established that the closed-loop dynamics are 


given by 
x A-—BG BG) x 
es 15.69 
2 | 0 A22 — aa | ( 


and hence that the eigenvalues of the closed-loop system are given by 
|sI —A+ BG||sI — Az2 + KAj2| = 0 (15.70) 


Thus, the separation principle also holds when a reduced-order observer is used. 
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It is important to recognize, however, that the separation principle applies only when the model of 
the process used in the observer agrees exactly with the actual dynamics of the physical process. It is not 
possible to meet this requirement in practice and, hence, the separation principle is an approximation 
at best. To assess the effect of a model discrepancy on the closed-loop dynamics, consider the following 
possibilities: 


Case 1: Error in dynamics matrix 
A=A+8A 


Case 2: Error in control distribution matrix 
B=B+8B 


Case 3: Error in observation matrix 
C=C+8C 


fi 


it is readily determined [6] that the characteristic polynomial of the complete, closed-loop system for 
cases 1 and 3 is given by 


Using the “metastate” 


sI—A¢ —BG 
I—A|= al, 15.71 
EPSON la sese | wh) 
where 
Ac=A-BG, A=A-KC 
Similarly, using the metastate 
H 
x= ; 
e 
it is found that the characteristic polynomial for case 2 is given by 
sI—A —8BG 
I—A|= ‘ 15.72 
ome. airy eo 


where 
A; =A—BG, A=A-—KC 


To assess the effect of perturbations of the dynamics matrices on the characteristic polynomial, the 
following determinantal identity can be used: 


2 | = |DI|A—cD7!B| (15.73) 


Apply Equation 15.73 to Equation 15.72 to obtain 


|sI — A| =|sI — A,||s! — A + 8BG(sI — A,)~!KC| 
= |sI — A,||s! — A]|I + 8BG(sI — A;)”!KC(sI — A)7}| 


(15.74) 


upon use of 


|AB] = |Al|B| 


The separation principle would continue to hold if the coefficient of 8B in Equation 15.74 were to 
vanish. It does vanish if observer matrix K satisfies the Doyle-Stein condition [2] 


K(I+C®K)"! = B(C®B)"!, (15.75) 
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where 
® =(s]— A)! 
is the plant resolvent. 
To verify this, note that 


(sl —A)~! =(sI -A+ KC)! =(@!+kc)"! 


(15.76) 
= &— ®K(I+C&K)'!Co@ 


When the Doyle-Stein condition (Equation 15.75) holds, Equation 15.76 becomes 
(sl — A)! = ® — ®B(CB) 'Ce, 


and so 
C(sI — A)" = Cb — C@B(CHB) Ce = 0, 
which ensures that the coefficient of 8B in Equation 15.74 vanishes and, hence, that the separation 
principle applies. 
Regarding the Doyle-Stein condition the following remarks are in order: 


¢ The Doyle-Stein condition can rarely be satisfied exactly. But, as shown [2], it can be satisfied 
approximately by making the observer a Kalman filter with a noise matrix of the form given by 
Equation 15.14. 

¢ The Doyle-Stein condition is not the only way the coefficient of 8B can vanish. However, the 
Doyle-Stein condition ensures other robustness properties. 

e An analogous condition for 8A and 8C can be specified. 


In carrying out a similar analysis for a reduced-order observer it is found that the characteristic 
polynomial for the closed-loop control system, when a reduced-order observer is used and the actual 
control distribution matrix B = B + 8B differs from the nominal (design) value B, is given by 


sI —F+ AG) AG 


fal = 
nee) BGs ae 


(15.77) 
where 
A = K8B, — By (15.78) 
It is seen that the characteristic polynomial of the closed-loop system reduces to that of Equation 15.70 
when 
A=0 (15.79) 
It is noted that Equation 15.79 can hold in a single-input system in which the loop gain is the only 
variable parameter. In this case 
8B, = pB,, 8Bz = pBz (15.80) 
and thus 


A = (KB, — Bz) = —pH 
Hence, if the observer is designed with 
H = B2 — KB; = 0, 


the separation principle holds for arbitrary changes in the loop gain. 
If Equation 15.79 cannot be satisfied, then, as shown in [7], condition analogous to the Doyle-Stein 
condition can be derived from Equation 15.77 in the case of a scalar control input (Equation 15.80): 


[I — K(I+ Aj. 22K) 'A12®22](B2 — KB1) = 0, (15.81) 


where 
y. = (sl — Az)! 
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15.6 Nonlinear Observers 


The concept of an observer carries over to nonlinear systems. However, for a nonlinear system, the 
structure of the observer is not nearly as obvious as it is for a linear system. The design of observers for 
nonlinear systems has been addressed by several authors, such as Thau [8] and Kou et. al. [9]. 

An observer for a plant, consisting of a dynamic system 


x = f(x, u) (15.82) 
with observations given by 
y=s(x,u) (15.83) 


is another dynamic system, the state of which is denoted by X, excited by the output y of the plant, having 
the property that the error 
e=x-Xx (15.84) 


converges to zero in the steady state. 
One way of obtaining an observer is to imitate the procedure used in a linear system, namely to 
construct a model of the original system (Equation 15.1) and force it with the “residual”: 


r=y—y=y—glj,u) (15.85) 
The equation of the observer thus becomes 
x= f(%,u)+K(y —g(%,u)), (15.86) 


where K() is a suitably chosen nonlinear function. (How to choose this function will be discussed later.) 
A block diagram representation of a general nonlinear observer is shown in Figure 15.3. 
The differential equation for the error e can be used to study its behavior. This equation is given by 
=k 
= f (x, u) — f(x, u) — K( g(x, u) — g(x, u)) (15.87) 
=f (x,u) —f(x —e,u)+k( g(x — e, u) — g(x, u)) 


Suppose that by the proper choice of k() the error Equation 15.87 can be made asymptotically stable, 
so that an equilibrium state is reached for which 


Plant model 


e=0 
y | fa 
OO «0 i—>C) - 
Observation _ | 
y ! fO | 
ae. | 
Control input ! 
a) | 


FIGURE 15.3 Structure of the nonlinear observer. 
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Then, in equilibrium, Equation 15.87 becomes 


0= f(x, u) — f(x —e,u) + K( g(x — e, u) — g(x, u)) (15.88) 


Since the right-hand side of Equation 15.88 becomes zero when e = 0, independent of x and u, it is 
apparent that e = 0 is an equilibrium state of Equation 15.87. This implies that if «() can be chosen to 
achieve asymptotic stability, the estimation error e converges to zero. 

It is very important to appreciate that the right-hand side of Equation 15.88 becomes zero independent 
of x and u only when the nonlinear functions f(-,-) and g(-,-) used in the observer are exactly the same 
as in Equations 15.82 and 15.83, which define the plant dynamics and observations, respectively. Any 
discrepancy between the corresponding functions generally prevents the right-hand side of Equation 15.88 
from vanishing and hence leads to a steady-state estimation error. Since the mathematical model of a 
physical process is always an approximation, in practice the steady-state estimation error generally does 
not tend to zero. But, by careful modeling, it is usually possible to minimize the discrepancies between the 
f and g functions of the true plant and the model used in the observer. This usually keeps the steady-state 
estimation error acceptably small. 

For the same reason that the model of the plant and the observation that is used in the observer must 
be accurate, it is important that the control u that goes into the plant is the very same control used in 
the observer. If the control to the plant is subject to saturation, for example, then the nonlinear function 
that models the saturation must be included in the observer. Failure to observe this precaution can cause 
difficulties. 

Including control saturation in the observer is particularly important as a means for avoiding the 
phenomenon known as integrator windup: the compensator, which has a pole at the origin, provides 
integral action. Imagine the transfer function of the compensator represented by an integrator in parallel 
with a second-order subsystem. The control signal to the integrator is oblivious to the fact that the input 
to the plant has saturated and hence keeps the integrator “winding up”; the error signal changes sign 
when the desired output reaches the set point, but the control signal does not drop from its maximum 
value. When the saturation is included in the observer, on the other hand, the control signal drops from 
its maximum value even before the error changes sign, thus correctly taking the dynamics (ie., the lag) 
of the process into account. 

The function k() in the observer must be selected to ensure asymptotic stability of the origin (e = 0 in 
Equation 15.88). By the theorem of Lyapunov’s first method (see Chapter 43), the origin is asymptotically 
stable if the Jacobian matrix of the dynamics, evaluated at the equilibrium state, corresponds to an 
asymptotically stable linear system. For the dynamics of the error Equation 15.87 the Jacobian matrix 
with respect to the error e evaluated at e = 0 is given by 


_(F\_ _(% 
A)=(4) x (3). (15.89) 


This is the nonlinear equivalent of the closed-loop observer equation of a linear system 
A, = A—KC, 


where A and C are the plant dynamics and observation matrices, respectively. The problem of selecting 
the gain matrix for a nonlinear observer is analogous to that of a linear observer, but somewhat more 
complicated by the presence of the nonlinearities that make the Jacobian matrices in Equation 15.89 
dependent on the state x of the plant. Nevertheless, the techniques used for selecting the gain for a linear 
observer can typically be adapted for a nonlinear observer. Pole placement is one method and the other 
is to make the observer an extended Kalman filter which, as explained later, entails online computation 
of the gains via the linearized variance equation. 

It should be noted that the observer closed-loop dynamics matrix depends on the actual state of the 
system and hence is time varying. The stability of the observer thus cannot be rigorously determined by 
the locations of the eigenvalues of A. 
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The choice of «(_) may be aided through the use of Lyapunov’s second method. Using this method, 
Thau [8] considered a “mildly nonlinear” process 


x =f (x) = Ax+wo(x), (15.90) 
where |1 is a small parameter, with linear observations 
y=Cx 
For this case, « can be simply a gain matrix chosen to stabilize the linear portion of the system 
K(r) = Kr, 


where K is chosen to stabilize 
A=A-KC 


This choice of K ensures asymptotic stability of the observer if () satisfies a Lipschitz condition 
Ild(u) — b(v)|| < kl lu— vI 


and when 
PA+A'P=-Q <—-cl 


In this case, asymptotic stability of the observer 
X = Ak + ro(X) + K(y — Cx) 
is assured for 


a) 
2k||P|| 


< 


This analysis was substantially extended by Kou et al. [9]. 
Suggestions for the choice of the nonlinear function k( ) have appeared in the technical literature. One 
suggestion [10], for example, is to use 


K(r) = W(%)! Kr, 


where K is a constant, possibly diagonal matrix, and W(x) is the Jacobian matrix defined by 


W(x) = ee) 
with 
Lyg(x) 
L2g(x) 
am=| |, 
Ljg(x) 


in which Ly g(x) is the nth Lie derivative of g(x) with respect to f(x). (See Chapter 38.) This approach 
can possibly be viewed as a nonlinear generalization of the Bass—Gura formula given previously, since the 
matrix W is akin to the observability matrix O defined earlier. 
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15.6.1 Using Zero-Crossing or Quantized Observations 


The ability of an observer to utilize nonlinear observations is not limited to observations that exhibit 
only moderate nonlinearity; even highly nonlinear observations can be accommodated. Perhaps the most 
nonlinear observation is that of a zero-crossing detector, in which 


ee 1, x>0 
dala ee ie 


This is the extreme special case of a quantizer, in which the output is quantized to only two levels. 
Suppose, for example, that the only observation is of the zerocrossing of x,. The observer for this 
process is then given by : 
x = f (x, u) — kLy — sgn(%1)], (15.91) 


as illustrated in Figure 15.4. 

Provided that a gain k, in this case a scalar parameter, can be found that stabilizes the observer at e = 0, 
the estimation error will be reduced to zero. The partial derivative of the nonlinear function with respect 
to the observation does not exist in this case because the observation is discontinuous with respect to 
the state. The stability of the observer cannot be established by linearizing about the origin. You have to 
use some other methods, such as Lyapunov’s second method, or determine the appropriate range of k by 
simulation. 

Some insight into how the observer operates can be gained by considering that both y and sgn(x;) are 
signals that take on the values of +1; their difference, which is the residual that appears in Equation 15.91, 
is either 0 or +2. Suppose the observer is working well; most of the time y and sgn(x;) will have the same 
sign and the residual will be zero. The residual will be nonzero for the short time interval in which y and 
sgn(x,) have different signs. The residual will thus consist of a train of narrow pulses, each of height +2 
and of width proportional to the phase difference between y and sgn(x;), as shown in Figure 15.5. The 
effect of each pulse is to nudge the state of the observer to agree with the state of the plant. 

The same idea extends to quantized observations, since a quantizer can be regarded as a multiple-level 
crossing detector. 

Of course, if there are other observations in addition to the zero-crossing observation, they can be 
combined with the latter and, with an appropriate choice of gains, can provide enhanced performance. 


15.6.2 Reduced-Order Observers 


Nonlinear reduced-order observers can be developed by the same methods that one uses for linear, 
reduced-order observers. 
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FIGURE 15.4 Observer using zero-crossing data. 
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FIGURE 15.5 Residual for zero-crossing observations consists of pulses. 


Suppose that the state is partitioned into two substates as given by Equation 15.25 with the observation 
given by 
y = g(x1,u) 


Provided that this expression can be solved for x; as a function of y and u, 


x=V(y,u)=y, 


we can use y as the observation. The completely general case in which y contains more state variables than 
its dimension can probably be handled in a manner similar to that used for linear systems, as discussed 
by Friedland [6]. The derivation is very tortuous in linear systems and is likely to be even more so in 
nonlinear systems and is, hence, omitted. 

Corresponding to the partitioning of the state vector x as in Equation 15.25, the dynamic equations are 
written as 


x1 = fi (x1, x2, u); (15.92) 
x2 = fo(x1, x2, u), (15.93) 


The nonlinear reduced-order observer is assumed to have the same structure as the corresponding 
linear observer. For the estimate of the substate x, we use the observation itself: 


R=y; (15.94) 
while the substate x is estimated using an observer of the form 
X2 = Ky +z, (15.95) 


where z is the state of a dynamic system of the same order as the dimension of the subvector x2 and is 
given by 
Z= o(y, Xp, u) (15.96) 


A block-diagram representation of the observer having the structure of Equations 15.94 through 15.96 
is given in Figure 15.6. 
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FIGURE 15.6 Reduced-order nonlinear observer. 


The object of the observer design is the determination of the gain matrix K and the nonlinear function 9. 
As for the full-order observer, these are to be selected such that 


¢ The steady-state error in estimating x2 converges to zero, independent of x and u. (The error in 
estimating x, is already zero when xX = y.) 
¢ The observer is asymptotically stable. 


As in the case of the full-order observer, we proceed by deriving the differential equation for the 
estimation error 
€=xX%.—-X2 (15.97) 


Using Equations 15.93, 15.95, and 15.92, we obtain 
@ =X) — X) = fal, x2, u) — Kfily, x2, u) — b(y, x2 — eu) (15.98) 


In order for the right-hand side of Equation 15.98 to vanish when e = 0, it is necessary that the function 
o(-,-, +) satisfy 
oy, x2, u) =frly, x2, u) — Kfi(y, x2, u) (15.99) 
for all values of y, x2, and u. 
To achieve asymptotic stability, the linearized system 


€= A(x2)e (15.100) 


Ata) = (32 = (2 «(F (15.101) 
Ox Ox Ox2 
must be asymptotically stable. 


As in the case of the full-order observer, there are several techniques for selecting an appropriate gain 
matrix. 

The reduced-order nonlinear observer can be further generalized by replacing the linear function Ky 
in Equation 15.95 by a nonlinear function k(y). 


with 


15.6.3 Extended Separation Principle 


When an observer having the structure described above is used to estimate the state of a linear system, 
and the estimate is used in place of the actual state, the poles of the closed-loop system comprise the poles 
of the observer and the poles that would be present if full-state feedback were implemented. This is the 


15-20 Control System Advanced Methods 


separation principle of linear systems. (But remember that this result holds only when the model of the 
plant used in implementing the observer is a faithful model of the physical plant.) 

The separation principle of linear systems can be extended to nonlinear systems. Consider, in particular, 
the nonlinear system 


k=f(x,u), (15.102) 


for which a control law 


u=y(x) (15.103) 


has been designed. Use of Equation 15.103 in Equation 15.102 gives the closed-loop dynamics 
x = f(x, y(x)) = F(x) (15.104) 


Assume that the closed-loop dynamics of the system with full-state feedback, as represented by F(x), 
has been designed—by whatever method might be appropriate—to achieve satisfactory behavior. How 
will the process behave when the state X of an observer is used in place of the true process state x [ie., 
when u = y(x)] ? As earlier, let 


X=x-e@, 
where ¢ is the error in estimating the state. Then Equation 15.103 becomes 
u = y(x) = y(x —e) 


Then Equation 15.104 becomes 
x =f (x, y(x — e)), (15.105) 


which together with Equation 15.87, becomes 
@=f(x, y(&)) — f(x — e, y&)) + KL g(x — e, y&)) — g(x YR), (15.106) 


which define the closed-loop dynamics. 

There is not much that can be done with Equations 15.105 and 15.106 when f, g, and y are general 
functions. However suppose these functions are sufficiently smooth to permit the use of Taylor’s theorem, 
that is, 


f(x, v(x — e)) =f (x, y(&) — (Of /Ay)(Oy/de)e + Ole’), (15.107) 
flx—ey(&)) =f (x, yR)) — (Of /Ax)e + O(e?), (15.108) 
g(x —e, y(X)) = g(x, y(&)) — (Og/x)e + Oe’), (15.109) 


where O(e*) represents terms that go to zero as |le||”. Then Equations 15.105 and 15.106 become 


x = F(x) — (Of /Oy)(Oy/de)e + Ole’), (15.110) 
é = [Of /Ax + K(Og/Ox)]e + Ole”) (15.111) 


With the terms of O(e”) omitted, a block-diagram representation of Equations 15.110 and 15.111 is 
shown in Figure 15.7. Note that the equation for the error has no input from the state estimation. The 
error thus converges asymptotically to zero as in the linear case. Since the error is the input to the full-state 
feedback control system, if the latter is asymptotically stable, the effect of the estimation error vanishes. 
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FIGURE 15.7 Illustration of the extended separation principle. 


15.6.4 Extended Kalman Filter 


Sometimes the nonlinear function k() can simply be a constant gain. Often, however, there is no obvious 
method of choosing this gain and a more systematic method is required. It is often appropriate to make 
the observer an “extended Kalman filter,” that is, to calculate the gain matrix online from the solution of 
the variance equation of the Kalman filter. 

Few of the many applications for which Kalman filters have been used have met the linearity require- 
ments of the theory. Nevertheless, the theory has been successfully, if not rigorously, applied. This is done 
by using a nonlinear observer of the form of Equation 15.88, but with the gain matrix K therein being 
computed, along with the state estimate, using Equations 15.12 and 15.13. The matrices A and C in these 
equations are the Jacobian matrices of the dynamics and observations 


A= [Of /Ox] =, (15.112) 
C = [0g/0x],—3, (15.113) 
for the nonlinear process defined by 
x =f (x, u), (15.114) 
y= Q(x, u) (15.115) 


In the linear case, the error covariance matrix, and through it the gain matrix K, does not depend on the 
estimated state. In principle, these matrices can be computed before the filter is implemented and stored 
in the filter’s memory. In the nonlinear case, however, the matrices A and C that are used in computing P 
and K depend on the state estimate. Hence, in the extended Kalman filter, the observer and the Kalman 
filter gain matrix computation are coupled. This means that the equations for both the variance equation 
and the observer must be implemented online as shown schematically in Figure 15.8. 

The requirement for online computation of the extended Kalman filter gain matrix can be a compu- 
tational burden. Even considering that the covariance equation P is symmetric, there still are k(k — 1)/2 
scalar differential equations in Equation 15.13 that must be integrated numerically in addition to the k 
scalar observer equations for a kth-order dynamic process. In a 10-order process, for example, a total 
of 55 equations must be integrated. It does not take a process of much higher order to overwhelm even 
a supercomputer. Moreover, the matrix Riccati equation (Equation 15.13) for P is notorious for being 
poorly behaved. Unless special measures are taken, the numerical solution to Equation 15.13 is likely to 
lose its positive-definite character as the theory requires. If this happens, the resulting state estimate x will 
probably be useless. 
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FIGURE 15.8 Schematic of the extended Kalman filter, showing coupling between state estimation and covariance 
computation. 


Fortunately, it is rarely necessary to be a stickler for accuracy in the implementation of Equation 15.13. 
In the first place, the entire theory of the extended Kalman filter is only approximate. Moreover, the 
spectral density matrices Q and R that appear in Equation 15.13 are hardly ever known to be better than 
an order of magnitude. Hence, any computational method that gives a reasonable approximation to P 
and K is usually acceptable. Some of the approximations that have been considered include the following: 


« Regard Pas being piecewise constant and compute it relatively infrequently, using the discrete-time 
version of the Kalman filter. 

¢ Simulate the observer and Equation 15.13 offline; examine the results and use appropriate approxi- 
mations. It may be possible, for example, to approximate some of the gains by constants. The effect 
of the approximations must be evaluated by further simulation. 

¢ Usea simpler model to represent the process in Equation 15.13 than is used as the process model in 
the observer; but be careful not to use an overly simple model in the implementation of the process 
dynamics. 
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Eigenstructure assignment is a useful tool that allows the designer to satisfy damping, settling time, and 
mode decoupling specifications directly by choosing eigenvalues and eigenvectors. Andry et al. [1] have 
applied eigenstructure assignment to design a stability augmentation system for the lateral dynamics of 
the L-1011 aircraft. Both constant gain output feedback and gain suppression designs are proposed. Sobel 
and Shapiro [30] used eigenstructure assignment to design dynamic compensators for the L-1011 aircraft. 
First- and second-order compensators were proposed for the case in which sideslip angle could not be 
measured. Later, Sobel et al. [33] proposed a systematic method for choosing the elements of the feedback 
gain matrix which can be suppressed to zero with minimal effect on the eigenvalue and eigenvector 


* Reprinted from Sobel, K.M., Shapiro, E.Y., and Andry, A.N., Jr., Int. J. Control, 59(1), 13-37, 1994. With permission [27]. 
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assignment. A design of an eigenstructure assignment gain-suppression flight controller is shown for 
F-18 High Angle of Attack Research Vehicle (HARV) aircraft. 

Specialized task tailored modes for highly maneuverable fighter aircraft have been designed by using 
eigenstructure assignment. Sobel and Shapiro [29] designed an eigenstructure assignment pitch pointing 
and vertical translation controller for the AFTI F-16 aircraft. Pilot command tracking was achieved by 
using a special case of O’Brien and Broussard’s [21] command generator tracker. Sobel and Shapiro [28] 
designed a yaw pointing and lateral translation controller for the linearized lateral dynamics of the Flight 
Propulsion Control Coupling (FPCC) aircraft. This conceptual control-configured vehicle has a vertical 
canard in addition to the more conventional control surfaces. 

This chapter is organized as follows. Section 16.2 describes eigenstructure assignment using constant 
gain output feedback. Section 16.3 extends eigenstructure assignment to allow the designer to suppress 
chosen elements of the feedback gain matrix to zero. A systematic method for choosing the entries to be 
suppressed is discussed. Section 16.4 describes eigenstructure assignment using dynamic compensation. 
Each section includes an application of eigenstructure assignment to the design ofa stability augmentation 
system for the linearized lateral dynamics of the F-18 HARV. Finally, in Section 16.5 we present a 
robust, sampled data, eigenstructure assignment control law design for the yaw pointing/lateral trans- 
lation maneuver of the FPCC aircraft. 


16.2 Eigenstructure Assignment Using Output Feedback 


Consider a system modeled by the linear time-invariant matrix differential equation described by 


x= Ax+Bu (16.1) 
y=Cx (16.2) 
where x is the state vector (m x 1), u is the control vector (m x 1), and y is the output vector (r x 1). It 
is assumed that the m inputs and the r outputs are independent. Also, as is usually the case in aircraft 


problems, it is assumed that m <r <n. If there are no exogenous inputs such as pilot commands, the 
feedback control vector u equals a matrix times the output vector y : 


u=—Fy (16.3) 
The feedback problem can be stated as follows: Given a set of desired eigenvalues, (a4), i=1,2,...,rand 
a corresponding set of desired eigenvectors, (v4), i= 1,2,...,r, find the real m x r matrix F such that 


the eigenvalues of A — BFC contain (#) as a subset, and the corresponding eigenvectors of A — BFC are 
close to the respective members of the set (vd ): 

Srinathkumar [34] has shown that if (A, B) is a controllable pair, then the feedback gain matrix F 
will exactly assign r eigenvalues. It will also assign the corresponding eigenvectors, provided that vd is 
chosen to be in the subspace spanned by the columns of (;J — A)~'B for i= 1,2,...,r. This subspace 
is of dimension m, which is the number of independent control variables. In general, a chosen or desired 
eigenvector vd will not reside in the prescribed subspace and, hence, cannot be achieved. Instead, a 
“best possible” choice for an achievable eigenvector is made. Andry et al. [1] showed that the best 
possible eigenvector is the projection of vd onto the subspace spanned by the columns of ();J — A)~!B. 
An alternative representation, described by Kautsky et al. [11], showed that the subspace in which the 
eigenvector v; must reside is also given by the null space of U; (\,I — A). The matrix U, is obtained from 
the singular-value decomposition of B, given by 


Au 
B=[Up, Uy] Be ie (16.4) 


The method of Kautsky et al. [11] for computing the subspaces is the preferred method for numerical 
computation. 
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Finally, the complete controllability assumption may be removed by using results derived by Liebst 
and Garrard [14] and by Liebst et al. [15]. These results allow the designer to alter eigenvectors that 
correspond to uncontrollable eigenvalues. 

In many practical situations, complete specification of v@ is neither required or known, but rather the 
designer is interested only in certain elements of the eigenvector. Thus, assume that v# has the following 
structure: 

vi = [vi XXX, X, Vij> Xx,X, vin] > 
where vj are designer-specified components and x is an unspecified component. Define, as shown by 
Andry et al. [1], a reordering operation { }* so that 


Ri el; 
‘| = |%*! 
vt = > (16.5) 
P= La] 
where @; is a vector of specified components of vd and dj is a vector of unspecified components of vi 


The rows of the matrix (d;1 —_A)~!B are also reordered to conform with the reordered components of 
d 
v?. Thus, 


{Qu — A)" 'B}* = Er (16.6) 


Then, as shown by Andry et al. [1], the achievable eigenvector v# is given by 
vf = (dil — A) BLT ;, (16.7) 


where (-)' denotes the appropriate pseudoinverse of (-). 
The output feedback gain matrix using eigenstructure assignment [1] is described by 


F=-—(Z—A,V)\(CV)"}, (16.8) 


where A, is the first m rows of the matrix A in Equation 16.1, V is the matrix whose columns are the r 
achievable eigenvectors, Z is a matrix whose columns are );z;, where the ith eigenvector ¥; is partitioned as 


1 


vj= with zj an m x 1 vector, and C is the output matrix in Equation 16.2. The result of Equation 16.8 


assumes that the system described by Equations 16.1 and 16.2 has been transformed into a system in which 
the control distribution matrix B is a lead block identity matrix. 
An alternative representation for the feedback gain matrix F developed by Sobel et al. [32] is 


F=—V,2,'Uj(VA—AV)V,2, Us, (16.9) 
where the singular-value decompositions of the matrices B and CV are given by 
xyV, 
B= (Ui0Unl| A ‘|. (16.10) 
yr at 
CV= (UoUnl| Aa (16.11) 
and where A is an r x r diagonal matrix with entries j,i = 1,2,...,r. The method described by Equa- 


tion 16.9 is the preferred method for numerical computation. 

We conclude the discussion with a comment about the closed-loop system stability. Unfortunately, it is 
not yet possible to ensure that stable open-loop eigenvalues do not move into the right half of the complex 
plane when an eigenstructure assignment output feedback controller is utilized. This is still an open area 
for further research. However, for aircraft flight control systems, the closed-loop stability requirement 
is neither necessary nor sufficient. For example, some modes, such as the dutch roll mode, are required 
to meet minimum frequency and damping specifications, as described in MIL-F-8785C [18]. For these 
modes, stability alone is not sufficient. Other modes, such as the spiral mode, may be unstable provided 
that the time to double amplitude is sufficiently large. For these modes, stability may not be necessary. 
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16.2.1 F-18 HARV Linearized Lateral Dynamics Design Example 


Consider the lateral directional dynamics of the F-18 HARV aircraft linearized at a Mach number of 
0.38, an altitude of 5000 ft, and an angle of attack of 5°. The aerodynamic model is augmented with 
first-order actuators and a yaw rate washout filter. The eight state variables are aileron deflection 8,, 
stabilator deflection 8;, rudder deflection 8,, sideslip angle f, roll rate p, yaw rate r, bank angle o, and 
washout filter state xg. The three control variables are aileron command 8,,, stabilator command 54,., 
and rudder command 8,,. The four measurements are fyo, p,B, and , where ryo is the washed out yaw 
rate. All quantities are in the body axis frame of reference with units of degrees or degrees/second. The 
state-space matrices A, B, and C that completely describe the model are shown in the Appendix. 

An output feedback gain matrix is now computed by using eigenstructure assignment. The desired 
dutch roll eigenvalues are chosen with a damping ratio of 0.707 and a natural frequency in the vicinity of 
3 rad/s. The roll subsidence and spiral modes are chosen to be merged into a complex mode, as suggested 
by [1]. The desired eigenvalues are 

Dutch roll mode: 


Roll mode: 


d ; 
droll =a £2 


The desired eigenvectors are chosen to keep the quantity |>/| small. Therefore, the desired dutch roll 
eigenvectors will have zero entries in the rows corresponding to bank angle and roll rate. The desired roll 
mode eigenvectors will have zero entries in the rows corresponding to yaw rate, sideslip, and xg (which is 
filtered yaw rate). The desired eigenvectors are 


x x x x71 84 
x x x x18 
x x x x | 8, 
1 x 0 0/18 

d . d . 

Var = 0 +) 0 Voll = 1 +j x |p . 
x 1 0 O]r 
(0) 0 x Ro) 
x x 0 0_] xg 


The achievable eigenvectors are computed by taking the orthogonal projection onto the null space 
of UP (iI — A). However, care must be taken when computing the pseudoinverse of L; because this 
matrix is ill-conditioned. Press et al. [24] suggest computing the pseudoinverse by using a singular-value 
decomposition in which the singular values, which are significantly smaller than the largest singular 
value, are treated as zero. The achievable eigenvectors given in this chapter were computed by using 
MATLAB® function PINV with TOL = 0.01. The achievable eigenvectors are shown in Table 16.1, 
where the underlined numbers indicate the small couplings between p, », and the dutch roll mode and 
between B, r, xg, and the roll mode. Hence, the ratio |/B| can be expected to be small. 

The feedback gain matrix is computed by using Equation 16.9 and is shown in Table 16.2. From the 
open-loop state responses to a 1° initial sideslip, shown in Figure 16.1, we conclude that the aircraft is 
poorly damped with strong coupling between the dutch roll mode and the roll mode. The closed-loop 
state response is shown in Figure 16.2. Observe that the maximum absolute values of the bank angle and 
roll rate are 0.0532 and 0.2819°/s., respectively. 

Finally, we consider the multivariable gain and phase margins for our design. Suppose that the modeling 
errors may be described by the matrix L given by 


L= Diag (C,e!*!, Le!%, ae stele) . 


Then, as shown by Lehtomaki [13], multivariable gain and phase margins at the inputs may be defined. 
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TABLE 16.1 Achievable Eigenvectors for Constant Gain Output Feedback 


Dutch Roll Mode Roll Mode 
1.1781 0.2929 0.3212 [ 0.2785 8a 
—2.7860 1.9039 0.3259 0.2058 8s 
10.1406 —4.1580 —0.1617 0.0693 & 
0.9777 ig 1.7260 —0.0558 re 0.0791 6 
—0.0559 oo —0.0094 0.9983 =o —4,9603 p 
5.7061 0.9932 —0.0016 —0.0246 t 
—0.0915 —0.1302 —0.9938 0.9916 o 
| —0.5258 | —1.0322 | —0.0022 | 0.0032 xg 
TABLE 16.2 Constant Gain Output Feedback Control Law 
Feedback Gain Matrix (Degree/Degree) . : 
Gain and Phase Margins max |6| 
Two Pp B (at inputs 8ac, 8sc. Src) max |p| 
—0.1704 0.1380 0.0277 
[—5.50 dB, 18.65 dB] 0.0532° 
0.7164 0.1075 —1.7252 
+52.41° 0.2819°/s. 
4.4961  —0.3877 


—2.2741 0.0173 
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Let Omin[I + FG(s)] > a, where the plant transfer matrix is given by G(s) = C(sI — A)~'B. Then, the 
upward gain margin is at least as large as 1/(1 — y) and the gain reduction margin is at least as small as 


1/(1+y). The phase margins are at least 4 


LSIN~!(y/2). The multivariable gain and phase margins for 


the constant gain output feedback design are shown in Table 16.2. We conclude that the margins are 
acceptable, especially because these margins are considered conservative. 


FIGURE 16.1 F-18 open-loop state responses. 
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FIGURE 16.2 F-18 closed-loop state responses for output feedback. 


16.3 Eigenstructure Assignment Using Constrained 
Output Feedback 


The feedback gain matrix given by Equation 16.8 feeds back every output to every input. We now consider 
the problem of constraining certain elements of F to be zero. By suppressing certain gains to zero, the 
designer reduces controller complexity and increases reliability. 

Using the development of [1], define 


Q=CV (16.12) 

W=Z—A\V (16.13) 
Then the expression for the feedback gain matrix F is given by (see Equation 16.8) 

F=-WQ'. (16.14) 


By using the Kronecker product and the lead block identity structure of the matrix B, each row of the 
feedback gain matrix can be computed independently of all the other rows [1]. Let y; be the ith row of 
the matrix WY. Then the solution for fj, which is the ith row of the feedback gain matrix F, is given by 


f=—WiQ, (16.15) 


If fi; is chosen to be constrained to zero, then Andry et al. [1] show that fj; should be deleted from fj and 
that the jth row of Q should be deleted. Let Q be the matrix Q with its jth row deleted and fj be the row 
vector fj with its jth entry deleted. Then, by using a pseudoinverse, the solution for fi, whose entries are 
the remaining active gains in the ith row of the matrix F, is given by 


fi=—vidh, (16.16) 


where (.)t denotes the appropriate pseudoinverse of (-). If more than one gain in a row of F is to be set to 
zero, the f; and Q must be appropriately modified. 
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16.3.1 Eigenvalue/Eigenvector Derivative Method for Choosing 
Gain-Suppression Structure 


Calvo-Ramon [5] has proposed a method for choosing a priori which gains should be set to zero based 
on the sensitivities of the eigenvalues to changes in the feedback gains. The first-order sensitivity of the 
hth eigenvalue to changes in the ijth entry of the matrix F is denoted by 0;,/Ofi. The expected shift in 
the eigenvalue };, when constraining feedback gain fij to zero is given by 


Si =()—: (16.17) 


Next, combine all the eigenvalue shifts that are related to the same feedback gain fi to form a decision 
matrix D* = {di}, D*eR™*", where 


i Be oa 1/2 
53] @)0)| oa 


and (-) denotes the complex conjugate of (-). 
The decision matrix D* is used to determine which feedback gains fj should be set to zero. If dj: is 


“small,” then setting fi; to zero will have a small effect on the closed-loop eigenvalues. Conversely, if dj: 
is “large,” then setting fj to zero will have a significant effect on the closed-loop eigenvalues. The control 
system designer must determine which di: are “small” and which are “large” for a particular problem. In 
this regard, it is assumed that the states, inputs, and outputs are scaled so that these variables are expressed 
in the same or equivalent units. 

The decision matrix D* was used by Calvo-Ramon [5] to design a constrained output feedback con- 
troller using eigenstructure assignment. However, the sensitivities of the eigenvectors with respect to the 
gains were not considered when deciding which feedback gains should be set to zero. Recall that the 
eigenvalues determine transient response characteristics such as overshoot and settling time, whereas 
the eigenvectors determine mode decoupling. This mode decoupling is related, for example, to the |/B| 
ratio in an aircraft lateral dynamics problem. This ratio must be small, as specified in [18], which implies 
that the closed-loop aircraft should exhibit a significant degree of decoupling between the dutch roll mode 
and the roll mode. The approach of Calvo-Ramon [5] may yield a constrained controller with acceptable 
overshoot and settling time, but the mode decoupling may be unacceptable. Thus, consideration of both 
eigenvalue and eigenvector sensitivities is important when choosing which feedback gains should be 
constrained to zero. 

Sobel et al. [33] extended the results of Calvo-Ramon [5] to include both eigenvalue and eigenvector 
sensitivities to the feedback gains. The eigenvector derivatives are used to compute the expected shift in 
eigenvector v; when constraining feedback gain fjj to be zero. This expected shift in eigenvector vp is 
given by 

3 = (Fi) (Ovn/ Of). (16.19) 


Then, all the eigenvector shifts related to the same feedback gain fj; are combined to form an eigenvector 


decision matrix D’, 
ile 1/2 
y ~h\* (xh 
DY = pz (5) ()| : (16.20) 
h=1 

where (-)* denotes the complex-conjugate transpose of (-). The gains that should be set to zero are 
determined by first eliminating those fj corresponding to entries of D* that are considered to be small. 
Then, those entries of D” corresponding to those fi that were chosen to be set to zero based on D* are 
reviewed. In this way, the designer can determine whether some of the fj that may be set to zero based 
on eigenvalue considerations should not be constrained based on eigenvector considerations. 
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16.3.2 F-18 HARV Linearized Lateral Dynamics Design Example 


We return to the example which was first considered in Section 16.2.1 and now we seek a constrained- 
output feedback controller. The eigenvalue decision matrix D* and the eigenvector decision matrix D” 
are shown in Table 16.3. The entries of D* considered large are underlined in Table 16.3. Observe 
that only seven of the 12 feedback gains are needed when only eigenvalue sensitivities are considered. 
The constrained-output feedback gain matrix based on using only the information available from the 
eigenvalue decision matrix D* is shown in Table 16.4. The state responses to a 6° initial sideslip are 
shown in Figure 16.3. Observe the significantly increased coupling between sideslip and bank angle as 
compared to the unconstrained design of Section 16.2.1. The maximum absolute values of the bank 
angle and roll rate are now 0.5102° and 1.7881°/s compared with 0.0532° and 0.2819°/s obtained with the 
unconstrained feedback gain matrix. The increased coupling is due to ignoring the eigenvector sensitivities 
and illustrates the importance of the eigenvectors in achieving adequate mode decoupling. 

Next, consider the entries of D” that correspond to those fjj that were chosen to be set to zero based 
on the eigenvalue decision matrix. The two largest di, that belong to this class are dy, and d},. A new 
constrained-output feedback gain matrix is computed in which f; and f)3 are not set to zero. Nine gains 
are now needed to be unconstrained when using both eigenvalue and eigenvector information. The new 
feedback gain matrix is given in Table 16.4 and the state responses for a 1° initial sideslip are shown in 
Figure 16.4. Observe that these time responses are almost identical to the responses in Figure 16.2, which 
were obtained by using all 12 feedback gains. Thus, a simpler controller is obtained with a negligible change 
in the aircraft time responses. Finally, the multivariable gain and phase margins are shown in Table 16.4. 
The values of these margins are considered acceptable because singular-value-based multivariable stability 
margin computation is conservative. 


TABLE 16.3 Eigenvalue and Eigenvector Decision Matrices 


Eigenvalue Decision D* Eigenvector Decision Matrix D” 
Two P B Two P B 
0.0710 0.5117 0.0019 0.2379 0.2379 0.4675 0.0107 0.1835 8a 
0.2286 0.3150 0.0843 0.3362 0.7927 0.2882 0.5270 0.1693 8s 
0.6754 0.0162 0.3347 0.0482 0.7780 0.0259 0.3194 0.0871 br 


TABLE 16.4 Comparison of Constant Gain Control Laws 


Feedback Gain Matrix Degree/Degree ; ; 
Gain and Phase Margins max || 


Two Pp 6 o (at inputs 84c, 8s. 8c) max |p| 


Unconstrained 

[—0.1704 0.1380 0.0277 0.4092 
0.7164 0.1075 —1.7252 0.4867 

| —-2.2741 0.0173 4.4961 —0.3877 

Constrained D* only 


[—5.50 dB, 18.65 dB] 0.0532° 
+52.41° 0.2819°/s 


T 0.0 0.1926 0.0 0.6417 
[—5.38 dB, 16.90 dB] —_-0.5102° 
0.3371 0.2010 0.0 0.7554 
+50.75° 1.7881°/s 
[2.2648 0.0 4.4891 0.0 
Constrained D* and D’ 
[—0.1643 0.1365 0.0 0.4049 


[—5.32 dB, 16.22 dB] 0.0505° 
+50.01° 0.2662°/s 


0.7164 0.1075 —1.7252 0.4867 
| —2.2648 0.0 4.4891 0.0 
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FIGURE 16.3 F-18 closed-loop state responses for constrained-output feedback (using only D*). 
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FIGURE 16.4 F-18 closed-loop state responses for constrained-output feedback (using both D* and D’). 
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16.4 Eigenstructure Assignment Using Dynamic Compensation 


We now generalize the eigenstructure assignment flight control design methodology to include the 
design of low-order dynamic compensators of any given order £,0 < €<n—r. Recall that n and r are 
the dimensions of the aircraft state and output vectors, respectively. Consider the linear time-invariant 
aircraft described by Equations 16.1 and 16.2 with a linear time-invariant dynamic controller specified by 


z(t) = Dz(t)+ Ey(t), (16.21) 
u(t) = F(z) + Gy(t), (16.22) 
where the controller state vector z(t) is of dimension /,0 < €<n-—r. 


It is convenient to model the aircraft and compensator by the composite system originally proposed 
by Johnson and Athans [10]. Thus, define 


x = Ax + Bi, (16.23) 
y= Cx, (16.24) 
u= Fy, (16.25) 
where 
2) oe) 2 TAY So PBT. o-" TON) 2 Fee 
z=[=], A= , B= , C= , F= , 
z 010 Ol Ol E|D 
Furthermore, the eigenvectors of the composite system may be described by 
1 Ea ; (16.26) 
vi(Z) 


where v;(x) is the ith subeigenvector corresponding to the aircraft and v;(z) the ith subeigenvector 
corresponding to the compensator. Once the compensator dimension is chosen, the problem is solved as 
previously shown for the constant gain case. 

The dynamic compensator design problem may be stated as follows. Given a set of desired aircraft 
eigenvalues {x7} ,i=1,2,...,r+€anda corresponding set of desired aircraft subeigenvectors v4 (x), i = 
1,2,...,r+4, find real matrices D(¢ x £), E(¢ x r), F(m x £), and G(m x r) so that the eigenvalues of 
A+ BFC contain {»¢ } as a subset and corresponding subeigenvectors {v;(x)} are close to the respective 
members of the set {v4 (x)}. 


16.4.1 F-18 HARV Linearized Lateral Dynamics Design Example 


We again return to the example first considered in Section 16.2.1. If y = [Two, p,B, 17, as was the case 
in Sections 16.2.1 and 16.3.1, then the designer can specify both the dutch roll mode and roll mode 
eigenvalues. The designer might also specify three entries in the real and imaginary parts of the dutch roll 
eigenvectors and four entries in the real and imaginary parts of the roll mode eigenvectors as was done in 
Section 16.2.1. Then achievable eigenvectors are computed. 

Now suppose that the measurement is given by y= [rwo,p, ]', but the designer is still required 
to assign both the dutch roll and roll mode eigenvalues. Using the results of Section 16.4, we might 
utilize a first-order dynamic compensator with state z;. The composite system has state vector x = 
[8a 855 875 Bs ps 1 Ps X85 z,]', and the measurement vector is given by y = [Two,p, O, z,]'. Thus, as before, 
the designer might choose to specify three or four entries of the real and imaginary parts of v;(x), i =1,2,3,4, 
which are the entries of the dutch roll and roll mode eigenvectors corresponding to the original aircraft 
state variables. Again, achievable eigenvectors will be computed. 


Eigenstructure Assignment 16-11 


We might ask whether the eigenvalue/eigenvector specifications are identical for both proposed prob- 
lems. Certainly, the eigenvalue specifications and the corresponding vd (x) subeigenvectors may be identi- 
cal. However, the vd (z) subeigenvector specifications must now be properly chosen. Otherwise, the modal 
matrix for the composite system may become numerically singular. 

Finally, we remark that if the first-order compensator does not perform acceptably, then the designer 
might try a higher-order compensator. In the case of a second-order compensator, the composite system 
has state vector x = [8,, 85, 8,58, p, 7, , Xg, Z1 z]', and the measurement vector is given by y = [Two p, 0; 
z1,Z2]'. In this case, the designer can also specify one of the compensator eigenvalues and some entries 
of its corresponding eigenvector. 

Now consider the case when only the washed-out yaw rate, roll rate, and bank angle are measured. 
We form the composite system described by Equations 16.23 through 16.25 by appending a first-order 
compensator to the aircraft dynamics. We specify that the roll mode and dutch roll mode eigenvalues 
are the same as in the constant gain feedback problem in Section 16.2.1. The compensator pole is not 
specified, but it is chosen by the eigenstructure assignment algorithm to obtain eigenvalue and eigenvector 
assignment for the aircraft modes. Furthermore, because we have three sensors plus one compensator 
state, the eigenstructure assignment algorithm will allow us to specify four closed-loop eigenvalues. We 
specify that the desired aircraft subeigenvectors v(x) are the same as the desired eigenvectors in the 
constant gain output feedback problem. The desired compensator subeigenvectors are chosen so that the 
dutch roll and roll modes participate in the compensator state solution. The control law is described by 


Sa 0.1708 —0.1383 —0.4079] [rwo —0.0040 
8; | = | —0.2723 —0.3328 —0.2408|| p | +] -1.4112| x, (16.27) 
Se 1.0897 0.5836 —1.5548] | o 3.7731 

2) = —2.4784z) +. 0.5906 ry + 0.9817p + 3.07960. (16.28) 


The desired eigenvalues, achievable eigenvalues, and desired eigenvectors are shown in Table 16.5. 
The separation principle does not apply to the dynamic compensator described by Equations 16.21 
and 16.22. Thus, the composite system has eigenvalues at — 1.7646 and —0.7549 which are due to the 
compensator and the yaw rate washout filter, respectively. The time responses of the aircraft states are 
shown in Figure 16.5 from which we observe that the responses with the dynamic compensator are slower 
than the responses with constant gain output feedback. However, the dynamic compensator controller 
is implemented without the need for a sideslip sensor. In addition, the ratio |/B| * 0.8 as compared to 
|b/B| ~ 4 for the open-loop aircraft. 


TABLE 16.5 Eigenvalues and Eigenvectors for Dynamic Compensator 


Desired Eigenvectors 


Dutch Roll Mode Roll Mode 


Desired Achievable 
Eigenvalues Eigenvalues Re Im Re Im 
[x [x [x [x 8a 
dp = —2 4 j2 x x % x 8s 
Ayoll = —3 + j2 x x x x 8 
act = —30.0000 x 1 0 0 
dp = —2 452 ae B 
act = —27.6993 0 0 x 1 p 
droll = —3 + j2 
act = —25.2634 1 x 0 0 r 
gine = —0.7549 0 0 1 x b 
comp = —1.7646 x x 0 0 xg 
[1 Lx. [1 Lx Z 


16-12 Control System Advanced Methods 


15 1 T 1 T 1 1 1 1 1 


am) 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 
Time (s) 


FIGURE 16.5 F-18 closed-loop state responses for first-order dynamic compensator. 


The multivariable stability margins are computed using oOmin[I + KG(s)] where G(s) = C(sI — A)'B 
and K = —[F(sI — D)~!E+G]. The multivariable gain margins are GMe[—2.47 dB, 3.46 dB ] and the 
multivariable phase margin is +18.89°. If the designer considers these margins inadequate, then an 
optimization may be used to compute an eigenstructure assignment controller with a constraint on the 


minimum of the smallest singular value of the return difference matrix. In the next section, we present a 
robust, sampled data eigenstructure controller for the yaw pointing/lateral translation maneuver of the 
FPCC aircraft. 


16.5 Robust, Sampled Data Eigenstructure Assignment 


Sobel and Shapiro [28] used eigenstructure assignment to design a continuous-time controller for the yaw 
pointing/lateral translation maneuver of the FPCC aircraft. This conceptual control-configured aircraft 
has a vertical canard and is difficult to control because the control distribution matrix has a minimum 
singular value of 0.0546. The design of Sobel and Shapiro [28] is characterized by perfect decoupling, 
but the minimum of the smallest singular value of the return difference matrix at the aircraft inputs was 
only 0.18. 

Sobel and Lallman [32] proposed a pseudocontrol strategy for reducing the dimension of the control 
space by using the singular-value decomposition. The FPCC yaw pointing/lateral translation design of 
Sobel and Lallman [32] yields a minimum of the smallest singular value of the return difference matrix at 
the aircraft inputs of 0.9835, but the lateral translation transient response has significant coupling to the 
heading angle. 

Sobel and Shapiro [31] have proposed an extended pseudocontrol strategy. Piou and Sobel [22] 
extended eigenstructure assignment to linear time-invariant plants which are represented by Middleton 
and Goodwin’s [17] unified delta model which is valid both for continuous-time and sampled data opera- 
tion of the plant. Piou et al. [23] have extended Yedavalli’s [35] Lyapunov approach for stability robustness 
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of a linear time-invariant system to the unified delta system. In this section, we design a robust, sampled 
data, extended pseudocontrol, eigenstructure assignment flight control law for the yaw pointing/lateral 
translation maneuver of the FPCC aircraft. The main goal of this section is to describe a design methodol- 
ogy which incorporates robustness into the eigenstructure assignment method. However, a sampled data 
design is used for illustration. 


16.5.1 Problem Formulation 


Consider a nominal linear time-invariant system described by (A, B, C). The corresponding sampled data 
system is described by (As, Bs, C) and the unified delta model is described by (Ap, By, C). Suppose that the 
nominal delta system is subject to linear time-invariant uncertainties in the entries of Ap, By described by 
dA, and dB,, respectively. Then, the delta system with uncertainty is given by (Ap + dAp, By + GBp, C). 
Here dAy=dA, dB, =4B in continuous time and dA, = dA3, dBy = dB in discrete time. Further- 
more, suppose that bounds are available on the maximum absolute values of the elements of dA and 
dB so that {dA : dA* < Amax} and {dB : dB* < Bmax} and where “<” is applied element by element to 
matrices. 

Consider the constant gain output feedback control law described by u(t) = Fpy(t), where Fp = F in 
continuous time and F, = Fs in discrete time. Then, the nominal closed-loop unified delta system is 
given by px(t) = Ap-x(t), where Ap- = A+ BFC in continuous time and As + BsFsC in discrete time. 
The uncertain closed-loop unified delta system is given by px(t) = Apcx(t) + dAp-x(t), where dAp, = 
dA + dB(FC) in continuous time and dA; + dB;(F;C) in discrete time. The reader is referred to Middleton 
and Goodwin [17] for a more detailed description. 


16.5.2 Pseudocontrol and Robustness Results 


The purpose of a pseudocontrol is to reduce the dimension of the control space. This reduction is needed 
for systems whose control distribution matrix B has a minimum singular value that is very small. After 
the eigenstructure assignment design is complete, the controller is mapped back into the original control 
space. Consider the singular-value decomposition of the matrix B, given by 


D] Ve 
By = [U, U2 Up] »») Vel (16.29) 
OI Vee 
where X) = diag [o1,...,0q] and Xz = diag [og41,...,0p] and where op < op_) < +--+ < Og41 <8 with 


€ small. 


Lemma 16.1: 


Let the system with the pseudocontrol u(t) be described by 


px(t) = Apx(t) + Byii(t), (16.30) 
y(t) = Cx(t), (16.31) 


where 


By = U1 + Un[ay, a2]. (16.32) 
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We design a feedback pseudocontrol for the system described by Equations 16.30 through 16.32. Then, 
the true control u(t) for the system described by (Ap, By, C) is given by 


u(t) = [yi=;" + V2=5'a| iu(t). (16.33) 


Furthermore, when a = [0,0]", the control law u(t) given by Equation 16.33 reduces to the control law 
given by Equation 20 in Sobel and Lallman [32]. 


Theorem 16.1: 


The system matrix Apc + dAp, is stable if 
ines ( Flees Bi sax), eal (16.34) 
where 
Ej) max = Ap max + Bpmax(FpC)* 
and 
Eo max = {In + A[Ap + Bo(FoC)]}* + (A/2)E1 max 
and where P, satisfies the Lyapunov equation given by 
AjcPp + PpApe + AApePp Age = —2In 


and where 1 is the matrix formed by the modulus of the entries of the matrix Po, and (-); denotes the 
symmetric part of a matrix. 


16.5.3 FPCC Yaw Pointing/Lateral Translation Controller Design Using 
Robust, Sampled Data, Eigenstructure Assignment 


We consider the FPCC aircraft linearized lateral dynamics which is described by Sobel and Lallman [32]. 
The state-space matrices A and B are shown in the Appendix. The state variables are sideslip angle B, bank 
angle @, roll rate p, and lateral directional flight path angle (y = ¥ + 8), where W is the heading angle. 
The control variables are rudder 8,, ailerons §,, and vertical canard 8,. The angles and surface deflections 
are in degrees, and the angular rates are in degrees/second. The five measurements are B, , p, 7, and y. 

First, we design an eigenstructure assignment control law by using an orthogonal projection. The delta 
state-space matrices As and Bs are computed by using the MATLAB Delta Toolbox. The sampling period 
A is chosen to be 0.02 s for illustrative purposes. The desired dutch roll, roll mode, and flight path mode 
eigenvalues are achieved exactly because five measurements are available for feedback. The achievable 
eigenvectors are computed by using the orthogonal projection of the ith desired eigenvector onto the 
subspace which is spanned by the columns of (y;I — As)~!Bs. The closed-loop delta eigenvalues yj, i = 
1,...,n, and the feedback gain matrix Fs are shown in Table 16.6. The desired closed-loop eigenvectors 
are shown in Table 16.7. 

The orthogonal projection solution is characterized by excellent decoupling with the minimum of the 
smallest singular value of (I + FG) equal to 0.18. Here the transfer function matrix of the delta plant is 
given by G([e/®4 — 1]/A), where 0 < w < 1/T. Furthermore, the Lyapunov robust stability condition of 
Equation 16.34 is not satisfied. 

To improve the minimum singular value of (J + FG), we design a controller by using an orthogonal 
projection with the pseudocontrol of Sobel and Lallman [32]. This pseudocontrol mapping is given 
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TABLE 16.6 Comparison of FPCC Designs (A = 0.02 s) 


Feedback Gain Matrix 

Closed-Loop Eigenvalues i) o P r y 
Orthogonal Vdr = —1.9990 +j1.921 1.4688 —0.2866 —0.0022 0.3799 —1.5332 &} 
projection design Yroll = —2.95 + j1.883 0.5652 —2.2990 —0.9044 —0,.6691 —0.8890 84 
Vip = 0.4975 9.2964 —1.2143 —0.514 —1.4219 —15.57 & 
Orthogonal Vdr = —1.999 £f1.921 —0.4929  —0.0332 0.0076 0.6883 2.7584 = &} 
projection design Yroll = —2.95 + j1.883 0.9517 2.353 —0.9093 —0.7565 —2.5147 84 
with pseudocontrol Vp = —0.4975 0.1203 —0.0530  —0.0245 —0.1535 —0.6023 & 
Robust V1,2 = —3.37 £j1.56 —0.2833 —0.0340 —0.0062 0.2377 0.5336 8 
pseudocontrol y3,4 = —3.16 £j1.473 1.5501 —2.4760 —1.0370 —0.9812 —1.7788 8&4 
design Vfp = —0.4310 5.6151  —0.0012  -—0.1486  -—4.6301 —10.285 %& 
Robust V1,2 = —2.55 £j1.19 —0.1658 —0.1967 —0.0808 0.4895 0.8616 8} 
pseudocontrol 3,4 = —2.60 471.22 0.8390 —1.5479 —0.7663 —0.8565 —1.2679 &4 
design with Vfp = —0.3248 1.1525  —0.8794 —0.4764 -—1.9712 —3.2578 & 


singular-value 


constraint 


Note: Eigenvalues are computed by using feedback gains with significant digits to machine precision. 


by Equation 16.33 with a = [0,0]. This design is characterized by a lateral translation response with 
significant coupling between y and W with the minimum of the smallest singular value of (I + FG) equal 
to 0.9835. The Lyapunov sufficient robust stability condition of Equation 16.34 is not satisfied. We note 
that the yaw pointing responses exhibit excellent decoupling for both orthogonal projection designs. 
Next, to obtain a robust design with excellent decoupling, we design a robust pseudocontrol law by 
using the design method proposed by Piou et al. [23]. This new design method minimizes an objective 
function which weights the heading angle due to a lateral flight path angle command and the lateral flight 
path angle due to a heading command. Constraints are placed on the time constants of the dutch roll, 
roll, and flight path modes, the damping ratios of the dutch roll and roll modes, and the new sufficient 
condition for robust stability. Mathematically, the objective function to be minimized is given by 


100 


is Ds, [a =o) (Dye +2 (VA ve] (16.35) 


k=1 


The upper limit on the index k is chosen to include the time interval kAg[0, 2] during which most of the 
transient response occurs. Of course, computation of Equation 16.35 requires that two linear simulations 
be performed during each function evaluation of the optimization. The constraints for continuous time 
and the corresponding constraints for discrete time are shown in Table 16.8 where ¢ is the damping ratio. 


TABLE 16.7 FPCC Desired Closed-Loop Eigenvectors 
Dutch Roll Mode Roll Mode Flight Path Mode 


x 1 0 0 x °) 
0 0 x 1 0 

0; +7) 0 Lj Sey] x 0 Pp 
1 x 0 0 0 r 
0 0 0 0 1] y 
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TABLE 16.8 Constraints for the FPCC Designs 


Continuous Time Discrete Time 


For Complex Eigenvalues: 
Red € [—4, -1.5] |1+Ayle [e*Ae1 A] 


—0.9b —0.4b 
¢ € [0.4,0.9] |l+Ayleé [exp (75) ,eXp (es) sh 
where $ = arg(1 + Ay) 


For the Real Eigenvalue: 
» € [-1, —0.05] [1+ Ay| € [e~4, e— 0-954] 
For Lyapunov Robustness: 
oy (FF ‘axPt EA max.) < 0.999 
For Multivariable Stability Margins (Final Design only): 
min Omin(I + FG) > 0.5550< ow < 1/T 


For illustrative purposes we have chosen Amax = 0.085A*t and Bmax = 0. After many trials, we found that 
a good value for the weight a in Equation 16.35 is a = 0.0075. 

The parameter vector contains the quantities which may be varied by the optimization. This 17- 
dimensional vector includes Re yq;, Im yar, Re Yrou» 1M You» Vg Re z1(1), Re 21 (2), Im z,(1), Im z(2), 
Re z3(1), Re z3(2), Im z3(1), Im z3(2), 25(1), 25(2), and the two-dimensional pseudocontrol vector a of 
Equation 16.32. Here, the two-dimensional complex vectors z; contain the free eigenvector parameters, 
that is, the ith eigenvector vj may be written as 


Yyi= LiZi, (16.36) 


where the columns of L; = (yjI — As)~!Bs are a basis for the subspace in which the ith eigenvector must 
reside. Thus, the free parameters are the vectors z; rather than the eigenvectors v;. The vectors z; are two 
dimensional because the optimization is performed in the two-dimensional pseudocontrol space. 

The optimization uses subroutine constr from the MATLAB Optimization Toolbox and subroutine 
delsim from the MATLAB Delta Toolbox. The optimization is initialized with the orthogonal projection 
pseudocontrol design which yields an initial value of 20.6 for the objective function of Equation 16.35 and 
a value of 1.66 for the right-hand-side (RHS) of the robustness condition of Equation 16.34. Unfortunately, 
the minimum of the smallest singular value of (J + FG) is only 0.2607 which is less than desired. 
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FIGURE 16.6 FPCC yaw pointing; robust pseudocontrol with singular-value constraint. 
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FIGURE 16.7 FPCC lateral translation; robust pseudocontrol with singular value constraint. 


To achieve a design with excellent time responses, Lyapunov robustness, and an acceptable minimum 
of the smallest singular value of (I + FG), we repeat the optimization with the additional constraint that 
Omin(I + FG) > 0.55. Once again we initialize the optimization at the orthogonal projection pseudocontrol 
design. The optimization yields an optimal objective function of 0.0556 and a value of 0.999 for the RHS 
of the robustness condition. The time responses for yaw pointing and lateral translation are shown in 
Figures 16.6 and 16.7, respectively. The lateral translation response is deemed excellent even though it has 
some small increase in coupling compared to the design without the additional singular-value constraint. 
Thus, we have obtained a controller which simultaneously achieves excellent time responses, Lyapunov 
robustness, and an acceptable minimum of the smallest singular-value of the return difference matrix at 
the aircraft inputs. 


16.6 Defining Terms 


Stability augmentation system: A feedback control designed to modify the inherent aerodynamic stabil- 
ity of the airframe. 

Gain suppression: A feedback control in which one or more entries of the gain matrix are constrained to 
be zero. 

Specialized task tailored modes: A feedback control system designed for a specific maneuver such as 
bombing, strafing, air-to-air combat, and so on. 

Mode decoupling: The elimination of interactions between modes, for example, in an aircraft mode 
decoupling is important so that a sideslip angle disturbance will not cause rolling motion. 

Eigenstructure assignment: A feedback control designed to modify both the system’s eigenvalues and 
eigenvectors. 
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Further Reading 


Kautsky et al. [11] suggested that eigenstructure assignment can be used to obtain a design with eigen- 
values, which are least sensitive to parameter variation, by reducing one of several sensitivity measures. 
Among these measures are the quadratic norm condition number of the closed-loop modal matrix and 
the sum of the squares of the quadratic norms of the left eigenvectors. Burrows et al. [4] have proposed a 
stability augmentation system for a well-behaved light aircraft by using eigenstructure assignment with an 
optimization which minimizes the condition number of the closed-loop modal matrix. Such an approach 
may sometimes produce an acceptable controller. In contrast to eigensensitivity, Doyle and Stein [7] 
have characterized the stability robustness of a multi-input, multi-output system by the minimum of the 
smallest singular value of the return difference matrix at the plant input or output. Gavito and Collins [8] 
have proposed an eigenstructure assignment design in which a constraint is placed on the minimum of 
the smallest singular value of the return difference matrix at the inputs of both an L-1011 aircraft and 
a CH-47 helicopter. Several authors have proposed different approaches to eigenstructure assignment. 
Clarke et al. [6] present a method to trade exact closed-loop eigenvalue location against an improvement 
in the associated eigenvector match. Bruyere et al. [3] use eigenstructure assignment in a polynomial 
framework. Satoh and Sugimoto [25] present a regional eigenstructure assignment using a linear matrix 
inequalities (LMI) approach. Nieto-Wire and Sobel [19] applied eigenstructure assignment to the design 
of a flight control system for a tailless aircraft. Other applications of eigenstructure assignment include 
air-to-air missiles [36], mechanical systems [26], and power systems [16]. Jiang [9], Konstantopoulos [12], 
and Ashari et al. [2] have proposed methods for reconfiguration using eigenstructure assignment. Nieto- 
Wire and Sobel [20] used eigenstructure assignment for the accomodation of symmetric lock in place 
actuator failures for a tailless aircraft. 


Appendix 


Data for the F-18 HARV Lateral Directional Dynamics at M = 0.38, H = 5000 ft., and a = 5° 


—30.0000 0 0 0 0 0 0 0 

0 —30.0000 0 0 0 0 0 0 

0 0 —30.0000 0 0 0 0 0 

_ | —0.0070 —0.0140 0.0412  —0.1727 0.0873 —0.9946 0.0760 0 
15.3225 12.0601 2.2022 —11.0723 —2.1912 0.7096 0 0 |’ 

—0.3264 0.2041 —1.3524 2.1137 —0.0086 —0.1399 0 0 

0 0 0 0 1.0000 0.0875 0 0 

0 0 0 0 0 0.5000 0 —0.5 


16-20 
30.0000 0 0 
0 30.0000 0 
0 0 30.0000 0 
0 0 0 0 
a 0 0 0 + 0 
0 0 0 0 
0 0 0 
0 0 0 


Data for the FPCC Lateral Directional Dynamics 


—0.340 0.0517 0.001 —0.997 0 

0 0 1 0 0 

A= |-2.69 0 —1.15 0.738 0 
5.91 0 0.138 —0.506 0 

—0.340 0.0517 0.001 0.0031 0 


oooco 
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0 0 0 1 0 -1 

0 0 1 0 0 +0 

0 10 00 0’ 

0 00 0 1 #40 
0.0755 0 0.0246 
0 0 0 

B= 4.48 5.22 —0.742 
—5.03 0.0998 0.984 
0.0755 0 0.0246 
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17.1 Introduction 


The linear quadratic regulator problem, commonly abbreviated as LQR, plays a key role in many control 
design methods. Not only is LQR a powerful design method, but in many respects it is also the mother of 
many current, systematic control design procedures for linear multiple-input, multiple output (MIMO) 
systems. Both the linear quadratic Gaussian, LQG or Hz, and Hoo controller design procedures have 
a usage and philosophy that are similar to the LQR methodology. As such, studying the proper usage 
and philosophy of LQR controllers is an excellent way to begin building an understanding of even more 
powerful design procedures. 

From the time of its conception in the 1960s, the LQR problem has been the subject of volumes of 
research. Yet, as will be detailed, the LQR is nothing more than the solution to a convex, least squares 
optimization problem that has some very attractive properties. Namely the optimal controller automati- 
cally ensures a stable closed-loop system, achieves guaranteed levels of stability robustness, and is simple 
to compute. To provide a control systems engineer with the knowledge needed to take advantage of 
these attractive properties, this chapter is written in a tutorial fashion and from a user’s point of view, 
without the complicated theoretical derivations of the results. A more in-depth reference that contains 
the theory for much of the material presented here is the text by Anderson and Moore [1], while the text 
by Kwakernaak and Sivan [2] is considered to be the classic text on the subject. 

In addition to the standard LQR results, robustness properties and useful variations of the LQR are 
also discussed. In particular, we describe how sensitivity weighted LQR (SWLQR) can be used to make 
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high-gain LQR controllers robust to parametric modeling errors, as well as how to include frequency 
domain specifications in the LQR framework through the use of frequency weighted cost functionals. 
Since this chapter can be viewed as an introduction to optimal control for MIMO, linear, time-invariant 
systems, we briefly discuss the full state feedback Ho controller and its relation to a mini-max quadratic 
optimal control problem for completeness. 


17.2 The Time-Invariant LQR Problem 


To begin, we simply state the LQR problem, its solution, and the assumptions used in obtaining the 
solution. The rest of this chapter is devoted to discussing the properties of the controller, how the 
mathematical optimization problem relates to the physics of control systems, and useful variations of 
the standard LQR. 


Theorem 17.1: Steady State LQR 


Given the system dynamics 
X(t) = Ax(t)+ Bu(t); x(t =0) = xo (17.1) 


with x(t) € R" and u(t) € R” along with a linear combination of states to keep small 
z(t) = Cx(t) (17.2) 


with z(t) € R?. We define a quadratic cost functional 


j= / - [27 z(t) + wT (Ru(s)| dt (17.3) 
0 


in which the size of the states of interest, z(t), is weighted relative to the amount of control action in u(t) 
through the weighting matrix R. 
If the following assumptions hold: 


1. The entire state vector x(t) is available for feedback 
2. [A B] is stabilizable and [A C] is detectable 
3. RR SO 


Then 
1. The linear quadratic controller is the unique, optimal, full state feedback control law 
u(t) = —Kx(t) withK =R7'B'S (17.4) 


that minimizes the cost, J, subject to the dynamic constraints imposed by the open-loop dynamics in 
Equation 17.1. 
2. Sis the unique, symmetric, positive semidefinite solution to the algebraic Riccati equation 


SA+A™S+C'C—SBR'B'S=0 (17.5) 
3. The closed-loop dynamics arrived at by substituting Equation 17.4 into Equation 17.1 
x(t) = [A — BK] x(t) (17.6) 


are guaranteed to be asymptotically stable. 
4. The minimum value of the cost J in Equation 17.3 is J = xd Sx. 
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While the theorem states the LQR result, it is still necessary to discuss how to take advantage of it. 
Before doing so, note that the control gains, K from Equation 17.4, can be readily computed for large-order 
systems using standard software packages such as MATLAB® and MATRIXx™. Hence, the remainder 
of this chapter is really about how the values of the design variables used in the optimization problem 
influence the behavior of the controllers. Here the design variables are z, the states to keep small, and R, 
the control weighting matrix. We begin the discussion by describing the physical motivation behind the 
optimization problem. 


17.2.1 Physical Motivation for the LQR 


The LQR problem statement and cost can be motivated in the following manner. Suppose that the system 
Equation 17.1 is initially excited, and that the net result of this excitation is reflected in the initial state 
vector xo. This initial condition can be regarded as an undesirable deviation from the equilibrium position 
of the system, x(t) = 0. Given these deviations, the objective of the control can essentially be viewed as 
selecting a control vector u(t) that regulates the state vector x(t) back to its equilibrium position of 
x(t) = 0 as quickly as possible. 

If the system Equation 17.1 is controllable, then it is possible to drive x(t) to zero in an arbitrarily short 
period of time. This would require very large control signals which, from an engineering point of view, 
are unacceptable. Large control signals will saturate actuators and if implemented in a feedback form will 
require high-bandwidth designs that may excite unmodeled dynamics. Hence, it is clear that there must 
be a balance between the desire to regulate perturbations in the state to equilibrium and the size of the 
control signals needed to do so. 

Minimizing the quadratic cost functional from Equation 17.3 is one way to quantify our desire to 
achieve this balance. Realize that the quadratic nature of both terms in the cost 


u'(t)Ru(t)>0 for u(t) 40 (17.7) 
z' (t)z(t) =x" (t)C'Cx(t)>0 for x(t) 40 (17.8) 


ensures that they will be non-negative for all ft. Further since the goal of the control law is to make the 
value of the cost as small as possible, larger values of the terms (Equations 17.7 and 17.8) are penalized 
more heavily than smaller ones. More specifically, the term (Equation 17.7) represents a penalty that helps 
the designer keep the magnitude of u(t) “small.” Hence the matrix R, which is often called the control 
weighting matrix, is the designer’s tool which influences how “small” u(t) will be. Selecting large values 
of R leads to small values of u(t), which is also evident from the control gain K given in Equation 17.4. 

The other term, Equation 17.8, generates a penalty in the cost when the states that are to be kept small, 
z(t), are different from their desired equilibrium value of zero. The selection of which states to keep small, 
that is the choice of C in Equation 17.2, is the means by which the control system designer communicates 
to the mathematics the relative importance of individual state variable deviations. That is, which errors 
are bothersome and to what degree they are so. 


Example 17.1: 


Consider the single degree of freedom, undamped, harmonic oscillator shown in Figure 17.1. Using 
Newton's laws, the equations of motion are 


mq(t) + kq(t) = u(t) (17.9) 


Letting x7(t) = q(t) denote the position of the mass and x(t) = g(t) its velocity, the dynamics 
(Equation 17.9) can be expressed in the state space as 


X(t) = x2(t) (17.10) 


X(t) = —w2x7 (t) + wu(t) 
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or 


with 


= X1(t) = 0 1 = 0 
He) AA eo2o|) = (or 


where w2 = 1/m is the square of the natural frequency of the system with k = 1. 

In the absence of control, initial conditions will produce a persistent sinusoidal motion of the mass 
at a frequency of w rad/sec. As such, we seek a control law that regulates the position of the mass to 
its equilibrium value of q(t) = 0. Thus, we define the states of interest, z, as x7 (t). 


z(t)=Cx(t) withC=[1 0] (17.11) 


Since the control is scalar, the cost (Equation 17.3) takes the form 
lee) 
j= i [20 a pu(t)| dt withp>0 
0 


where is the control weighting parameter. 

The optimal LQR control law takes the form u(t) = —Kx(t) where the LQ gain K is a row vector 
K= [ky ky]. To determine the value of the gain, we must solve the algebraic Riccati equation 17.5. 
Recalling that we want a symmetric solution to the Riccati equation, letting 


S14 a 
S= 17.12 
B S22 ( 


and using the values of A, B, and C given above, the Riccati equation 17.5 becomes 
— {S11 S12 0 1 0 —w2) [511 S12 1 
= a 592}|—w2 0 “ 1 0 Si2 $22 a 0 [} 0] 
1] Si ul | "| 2 BE | 
a 0 
0 ee S22 | | w? oe] S12 S22 


Carrying out the matrix multiplications leads to the following three equations in the three unknown 
Sij from Equation 17.12. 


1 
0 = 2w*5q2 — 14 -w454, (17.13) 
p 

2 1 4 

0=—S$1; + > ar $12522 (17.14) 
1 

0 = —2512 + - "55, (17.15) 
p 


Solving Equation 17.13 for S12 and simplifying yields 


. —ptp/1+1/p 
a” a 


Both the positive and negative choices for S$; are valid solutions of the Riccati equation. While we 
are only interested in the positive semidefinite solution, we still need more information to resolve 
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which choice of S12 leads to the unique choice of S$ > 0. Rewriting Equation 17.15 as 
Lae 
2512 = —@ ‘S55 (17.16) 
p 


indicates that 5; must be positive to satisfy the equality of Equation 17.16, since the right-hand side 
of the equation must always be positive. Equation 17.16 indicates that there will also be a + sign 
ambiguity in selecting the appropriate 53. To resolve the ambiguity we use Sylvester's Test, which 
says that for S > 0 both 

$11 =>0 and $11$22 — 54 > 0 (17.17) 


Solving Equation 17.15 and 17.13 using the relations in Equation 17.17, which clearly show that 
S22 > 0, gives the remaining elements of S 


ope © faa +1/0)(\/1+1/e-1) 
—p+p/1+1/p 
2 
(69) 


Si2= 


= 5 2(\/1+1/e-1v) 


The final step in computing the controller is to evaluate the control gains K from Equation 17.4. 


Doing so gives 
Kh=—irijpat B= —[2(\/1+ 1/61) 


Given the control gains as a function of the control weighting p it is useful to examine the locus of 
the closed-loop poles for the system as p varies over 0 < p < oo. Evaluating the eigenvalues of the 
closed-loop dynamics from Equation 17.6 leads to a pair of complex conjugate closed-loop poles 


mae 2 y(t +1/o~1) £32,)2(/1+1/0-+1). 


A plot of the poles as p varies over 0 < p < oo is shown in Figure 17 2. Notice that for large values 
of p, the poles are near their open-loop values at +7, and as p decreases the poles move farther out 
into the left half-plane. This is consistent with how p influences the cost. Large values of p place a 
heavy penalty on the control and lead to low gains with slow transients, while small values of p tell 
the mathematics that large control gains with fast transients are acceptable. 


17.2.2 Designing LOR Controllers 


While the above section presents all the formulas needed to start designing MIMO LQR controllers, the 
point of this section is to inform the potential user of the limitations of the methodology. The most 
restrictive aspects of LQR controllers is that they are full-state feedback controllers. This means that every 
state that appears in the model of the physical system Equation 17.1 must be measured by a sensor. In 
fact, the notation of z(t) = Cx(t) for the linear combination of states that are to be regulated to zero is 
deliberate. We do not call z(t) the outputs of the system because all the states x(t) must be measured in 
real time to implement the control law Equation 17.4. 

Full-state feedback is appropriate and can be applied to systems whose dynamics are described by a 
finite set of differential equations and whose states can readily be measured. An aircraft in steady, level 
flight is an example of a system whose entire state can be measured with sensors. In fact, LQR control has 
been used to design flight control systems for modern aircraft. 
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FIGURE 17.2 Locus of closed-loop pole locations for the single degree of freedom oscillator with w = 1 as ¢ varies 
over0< p< OO. 


On the other hand, full-state feedback is typically not appropriate for flexible systems. The dynamics 
of flexible systems are described by partial differential equations that often require very high-order, if not 
infinite-dimensional, state-space models. As such, it is not feasible to measure all the states of systems 
that possess flexible dynamics. Returning to the aircraft example, one could not use a LQR controller to 
regulate the aerodynamically induced vibrations of the aircraft’s wing. The number of sensors that would 
be needed to measure the state of the vibrating wing prohibit this. 

Having discussed the implications of full-state feedback, the next restrictive aspect of LQR to discuss 
is the gap between what the LQR controller achieves and the desired control system performance. Recall 
that the LQR is the control that minimizes the quadratic cost of Equation 17.3 subject to the constraints 
imposed by the system dynamics. This optimization problem and the resulting optimal controller have 
very little to do with more meaningful control system specifications like levels of disturbance rejection, 
overshoot in tracking, and stability margins. This gap must always be kept in mind when using LQR to 
design feedback controllers. The fact that LQR controllers are in some sense optimal is of no consequence 
if they do not meet the performance goals. Further since control system specifications are not given in 
terms of minimizing quadratic costs, it becomes the job of the designer to use the LQR tool wisely. To 
do so it helps to adopt a means-to-an-end design philosophy where the LQR is viewed as a tool, or the 
means, used to achieve the desired control system performance, or the end. 

One last issue to point out is that LQR controller design is an iterative process even though the 
methodology systematically produces optimal, stabilizing controllers. Since the LQR formulation does 
not directly allow one to achieve standard control system specifications, trial and error iteration over the 
values of the weights in the cost is necessary to arrive at satisfactory controllers. Typically LQR designs 
are carried out by choosing values for the design weights, synthesizing the control law, evaluating how 
well the control law achieves the desired robustness and performance, and iterating through this process 
until a satisfactory controller is found. In the sequel, various properties and weight selection tools will 
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be presented that provide good physical and mathematical guidance for selecting values for the LQR 
design variables. Yet these are only guides, and as such they will not eliminate the iterative nature of LQR 
controller design. 


17.3 Properties of LQR 


The LQR has several very important properties, which we summarize below. It is important to stress that 
the properties of LQR designs hinge upon the fact that full-state feedback is used and the specific way that 
the control gain matrix K is computed from the solution of the Riccati equation. 


17.3.1 Robustness Properties 


To visualize the robustness properties of LQR controllers, it is necessary to consider the loop transfer 
function matrix that results when implementing the control law of Equation 17.4. The LQR loop transfer 
function matrix, denoted by G;Q(s), induced by the control scheme of Equation 17.4 is given by 


Gio(s) = K(sI — A)'B 


and the closed-loop dynamics of Equation 17.6 using this representation are shown in Figure 17.3. An 
interesting fact is that GzQ(s) is always square and minimum phase. Note that as a consequence of this 
feedback architecture any unstructured modeling errors must be reflected to the inputs of the LQR loop, 
location 1 in Figure 17.3. 

Under the assumption that the control weight matrix R = R’ > 0 is diagonal, the LQR loop transfer 
matrix is guaranteed to satisfy both of the inequalities 


Omin [I + Gra¥o)] 21 
1 (17.18) 
- Vo 

2, 


Omin [I+ Graig) *] et 


where Omin denotes the minimum singular value. Since the multivariable robustness properties of any 
design depend on the size of Omin [I + G(yw)] and omin [I + G( jo) *] the following guaranteed multi- 
variable gain and phase margins are inherent to LQR controllers as a result of Equation 17.18. 


17.3.1.1 LQR Stability Robustness Properties 


1. Upward gain margin is infinite 
2. Downward gain margin is at least 1/2 
3. Phase margin is at least +60° 


These gain and phase margins can occur independently and simultaneously in all m control channels. To 
visualize this, consider Figure 17.4, where the f;(-) can be viewed as perturbations to the individual inputs, 
uj = fi(-)yi. As a result of the gain margin properties (1) and (2), the LQR system shown in Figure 17.4 


x(s) —u(s) 
x>| B , (sI-A)"1 > K > 
1 


FIGURE 17.3. The LQR loop transfer matrix, G_g(s) = K(sI — A)!B. 
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FIGURE 17.4 LQR loop used to visualize the guaranteed robustness properties. 


is guaranteed to be stable for any set of scalar gains B; with fj = B; where the f; lie anywhere in the 
range 1/2 < B; < oo. The phase margin property (3) ensures the LQR system shown in Figure 17.4 is 
guaranteed to be stable for any set of scalar phases ¢; with f; = e/®' where the ¢; can lie anywhere in the 
range —60° < 9; < +60°. 

These inherent robustness properties of LQR designs are useful in many applications. To further 
appreciate what they mean, consider a single input system with a single variable we wish to keep small 


X(t) = Ax(t)+ bu(t) with z(t) = c’ x(t) 


and let the control weight be a positive scalar, R= p > 0. Then the resulting LQR loop transfer function 
is scalar, and its robustness properties can be visualized by plotting the Nyquist diagram of Gza(jw) for 
all w. Figure 17.5 contains a Nyquist plot for a scalar G_q(jw) that illustrates why the LQR obtains good 
gain and phase margins. Essentially, inequality Equation 17.18 guarantees that the Nyquist plot will not 
penetrate the unit circle centered at the critical point at (—1, 0). 


17.3.2 Asymptotic Properties of LQR Controllers 


It is clear that the closed-loop poles of the LQR design will depend upon the values of the design parameters 
z(t) and R. The exact numerical values of the closed-loop poles can only be determined using a digital 
computer, since we have to solve the Riccati equation 17.5. However, it is possible to qualitatively predict 
the asymptotic behavior of the closed-loop poles for the LQR as the size of the control gain is varied 
without solving the Riccati equation, evaluating the control gains, and computing the closed-loop poles. 


A Imaginary 
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FIGURE 17.5 A typical Nyquist plot for a single-input, single-output LQR controller. 
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17.3.2.1 Assumptions for Asymptotic LQR Properties 


1. The number of variables to keep small is equal to the number of controls. That is dim[z(t)] = 
dim[u(t)] = m. 

2. The control weight is chosen such that R = pR where p is a positive scalar and R = R? > 0. 

3. G,(s) is defined to be the square transfer function matrix between the variables we wish to keep 
small and the controls with the loop open, z(s) = Gz(s)u(s) where G;(s) = C(sI — A)~'B, and qis 
the number of transmission zeros of G;(s). 


Adjusting p directly influences the feedback control gain. When p — ov, we speak of the LQR controller 
as having low gain since under Assumption (2) 


i 
u(t) = —-—R~'B! Sx(t) > 0 
Q 


as p —> 00. Likewise, when p — 0 we speak of the high gain controller since u(t) will clearly become large. 
The asymptotic properties of the LQR closed-loop poles under the stated assumptions are as follows. 


17.3.2.2 LQR Asymptotic Properties 


1. Low Gain: As p — on, the closed-loop poles start at 
a. The stable open-loop poles of G,(s) 
b. The mirror image, about the jw-axis, of any unstable open-loop poles of G,(s) 
c. If any open-loop poles of G(s) lie exactly on the jw-axis, the closed-loop poles start just 
to the left of them 
2. High Gain: As p — 0, the closed-loop poles will 
a. Go to cancel any minimum phase zeros of G,(s) 
b. Go to the mirror image, about the y-axis, of any non-minimum phase zeros of G;(s) 
c. The rest will go off to infinity along stable Butterworth patterns 


Realize that these rules are not sufficient for constructing the entire closed-loop pole root locus. 
However, they do provide good insight into the asymptotic behavior of LQR controllers. In particular, the 
second rule highlights how the choice of z(t), which defines the zeros of G,(s), influences the closed-loop 
behavior of LQR controllers. 

Extensions of these results exist for the case where Assumption (1) is not satisfied, that is when 
dim[z(t)] 4 dim[u(t)]. When dim[z(t)] 4 dim[u(t)], the low gain asymptotic result will still hold, which 
is to be expected. The difficulty for non-square G;(s) arises in the high-gain case as p — 0 because it is 
not a simple manner to evaluate the zeros that the closed-loop poles will go toward. For details of how to 
evaluate the zero locations that the closed-loop poles will go toward in the cheap control limit, one can 
refer to [3] and [2, problem 3.14]. 


17.4 LQR Gain Selection Tools 


In this section we present some variations on the standard LQR problem given in Theorem 17.1. It is best 
to view the following variations on LQR as tools a designer can use to arrive at controllers that meet a set 
of desired control system specifications. While the tools are quite useful, the choice of which tool to use 
is very problem specific. Hence, we also present some physical motivation to highlight when the various 
tools might be useful. Realize that these tools do not remove the iterative nature of LQR control design. 
In fact, these tools are only helpful when used iteratively and when used with an understanding of the 
underlying physics of the design problem and the limitations of the design model. The variations on LQR 
presented here are by no means a complete list. 
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17.4.1 Cross-Weighted Costs 


Consider the linear time-invariant system with dynamics described by the state equation in Equation 17.1 
and the following set of variables we wish to keep small 


z(t) = Cx(t) + Du(t) (17.19) 


Substituting Equation 17.19 into the quadratic cost from Equation 17.3 produces the cost function 
CO 
pe / {xT (NCT cx(t) 42x" (t)CT Du(t) + u! (t) [R gs p'p| u(t) dt (17.20) 
0 


which contains a cross penalty on the state and control, 2x! (t)C! Du(t). The solution to the LQR controller 
that minimizes the cost of Equation 17.20 will be presented here, but first we give some motivation for 
such a cost and choice of performance variables. 

Cross-weighted cost functions and performance variables with control feed-through terms are com- 
mon. The control feed-through term, Du(t) in Equation 17.19, arises physically when the variables we 
wish to keep small are derivatives of the variables that appear in the state vector of the open-loop dynamics 
Equation 17.1. To see this, consider the single degree of freedom oscillator from Example 17.1 where the 
acceleration of the mass is the variable we wish to keep small, z(t) = q(t). Using the definitions from the 
example and Equations 17.11, this can be expressed as 


z(t) = q(t) = x2(t) 
= —0"x,(t) + w7u(t) =[—w* O]x(t) + w u(t) 


Clearly, penalizing acceleration requires the control feed-through term in the definition of the variables 
we wish to keep small. Control feed-through terms also arise when penalizing states of systems whose 
models contain neglected or unknown, higher-order modes. Such models often contain control feed- 
through terms to account for the low-frequency components of the dynamics of the modes that are not 
present in the design model. Further, as will be seen below, cross-coupled costs such as Equation 17.20 
arise when frequency-dependent weighting terms are used to synthesize LQR controllers. 


Theorem 17.2: Cross-Weighted LQR 


Consider the system dynamics 
x(t) = Ax(t)+Bu(t) x(t=0)=xo (17.21) 


with x(t) € R" and u(t) € R” and the quadratic cost with a cross penalty on state and control 
[o,@) 
jx / [+7 ORexx(t) 2x7 (t)Ryu(t) + u™ (Rutt) dt (17.22) 
0 


in which the size of the states is weighted relative to the amount of control action in u(t) through the state 
weighting matrix Rx x, the control weighting matrix Ruy, and the cross weighting matrix Rxu. 
If the following assumptions hold: 


1. The entire state vector x(t) is available for feedback 


Rex R 
2: Rea RESO Ry SRS 0 and | oy | 
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1/2 
XX 


3. [A B] is stabilizableand[A  R,{“] is detectable* 


Then 


1. The linear quadratic controller is the unique, optimal, full-state feedback control law 
u(t)=—Kx(t) with K = Rj) (Ri, +B°S) (17.23) 


that minimizes the cost, J of Equation 17.22, subject to the dynamic constraints imposed by the 
open-loop dynamics in Equation 17.21. 

2. Defining Ay = (A —BR,,|R1,), S is the unique, symmetric, positive semi-definite solution to the 
algebraic Riccati equation 


SAy + AlS+ (Ra - RuuRvat Riu) — SBR;,| B'S =0 (17.24) 


3. The closed-loop dynamics arrived at by substituting Equation 17.23 into Equation 17.21 are guaran- 
teed to be asymptotically stable. 


Note that when the cost is defined by Equation 17.3 and the variables to be kept small are defined 
by Equation 17.19, Ry. = C!C, Rxy = C'D, and Ry, = [R + D'D| as can be seen from Equation 17.20. 
Also, if Rx, = 0 this theorem reduces to the standard LQR result presented in Theorem 17.1. 

The assumptions used in Theorem 17.2 are essentially those used in the standard LQR result. One 
difference is that here we require Rx = R2, > 0, while in Theorem 17.1 this was automatically taken care 
of by our definition of the state cost as x! (t)C! Cx(t) because C'C > 0. Unfortunately, the guaranteed 
robustness properties presented in the previous section are not necessarily true for the LQR controllers 
that minimize the cost (Equation 17.22). 


17.4.2 The Stochastic LQR Problem 


In the stochastic LQR problem, we allow the inclusion of a white noise process in the state dynamics. 
Namely, the state vector x(t) now satisfies the stochastic differential equation 


X(t) = Ax(t) + Bu(t) + LE(t) (17.25) 


where &(f) is a vector valued, zero-mean, white noise process. We still assume that all the state variables 
can be measured exactly and used for feedback as necessary. Physically, the term L(t) in Equation 17.25 
should be viewed as the impact of a persistent set of disturbances on a system that corrupt the desired 
equilibrium value of the state, x(t) = 0. The L matrix describes how the disturbances impact the system, 
and the stochastic process &(¢) captures the dynamics of how the disturbances evolve with time. In 
particular, since the disturbances are a white noise process, they are completely unpredictable from 
moment to moment. 

As with the standard LQR problem, we would like to find the control law that minimizes some quadratic 
cost functional which captures a trade off between state cost and control cost. However, the cost functional 
for the standard LQR problem, Equation 17.3, cannot be used here as a result of the stochastic nature of 
the states. The value of the cost functional would be infinity because of the continuing excitation from 
&(t). To accommodate the stochastic nature of the state, we seek the optimal control that minimizes the 


1/2. : ‘ : ne aguas is . 
i Rif is the matrix square root of Rxx which always exists for positive semidefinite matrices. For the cross-coupled cost in 


Equation 17.20, Rx, = C?C and Re? ='C, 
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expected value, or mean, of the standard LQR cost functional 


TOOT 


jan lim -[ [27 elt) + wT (Ru(s)| ar} (17.26) 
0 


where E is the expected value operator. The normalization by the integration time t is necessary to ensure 
that the cost functional is finite. Surprisingly, the optimal control for this stochastic version of the LQR 
problem is identical to that of the corresponding deterministic LQR problem. In particular, Theorem 17.1 
also describes the unique optimal full-state feedback control law that minimizes the cost Equation 17.26 
subject to the constraints Equation 17.25. This is a result of the fact that &(t) is a white noise process. Since 
the disturbances are completely unpredictable, the optimal thing to do is ignore them. As a consequence 
of all this, the properties of the deterministic LQR problem from Theorem 17.1 discussed in Section 17.3 
are also true for this stochastic LQR problem. 

Unfortunately, these results do not make it any easier to design LQR controllers to specifically reject 
disturbances that directly impact the state dynamics. This is because physical disturbances are typically 
not white in nature, and the stochastic LQR result only applies for white noise disturbances. However, 
these results do play a key role in synthesizing multivariable controllers when some of the states cannot 
be measured and fed back for control. 


17.4.3 Sensitivity Weighted LQR 


Synthesizing LQR controllers requires an accurate model of the system dynamics. Not only are the control 
gains, K, a function of the system matrices A and B, but in the high-gain limit we know that the closed- 
loop poles will go to cancel the zeros of G,(s). If the models upon which the control design is based are 
inaccurate, the closed-loop system at a minimum will not achieve the predicted performance and will, in 
the worst case, be unstable. This is particularly relevant for systems with lightly damped poles and zeros 
near the jw axis. Hence, it is important to address the issue of stability robustness, since designers often 
must work with models that do not capture the dynamical behavior of their systems exactly. 

Modeling errors can be classified into two main groups: unmodeled dynamics and parametric errors. 
Unmodeled dynamics are typically the result of unmodeled non-linear behavior and neglected high- 
frequency dynamics which lie beyond the control bandwidth of interest. How one can design LQR 
controllers that are robust to unmodeled dynamics will be discussed in the following section on frequency 
weighted LQR. Parametric uncertainty arises when there are errors in the values of the system parameters 
used to form the model. Sensitivity weighted LQR (SWLQR), is a variation on the standard LQR problem 
that can be used to increase the stability robustness of LQR controllers to parametric modeling errors. 
Before presenting the SWLQR results, we will discuss parametric modeling errors. 


17.4.3.1 Parametric Modeling Errors 


When parametric errors exist, the model will be able to capture the system’s nominal dynamics, but the 
model cannot capture the exact behavior. Parametric errors occur often because it is difficult to know the 
exact properties of the physical devices that make up a system and because the physical properties may 
vary with time or the environment in which they are placed. For example, consider the single degree of 
freedom oscillator from Example 17.1 as the model of a car tire. While we might have some knowledge 
of the tire’s stiffness, it is unrealistic to expect to know its exact value, as the tire is a complex physical 
device whose properties change with wear and temperature. Hence, the value of the spring stiffness k, 
which models the tire’s elasticity, will be parametrically uncertain. 

To deal with any sort of modeling error, one must formulate a model of the uncertainty in the nominal 
design model. Parametrically uncertain variables may appear directly as the elements of, or be highly 
nonlinear functions of, the elements of the A and B matrices. When the latter is the case, one often 
reduces the uncertainty model to parametric uncertainty in more fundamental system quantities such 
as the frequency and/or damping of the system’s poles and zeros. In either case one must decide which 
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elements of the model, be they pole frequencies or elements of the A matrix, are uncertain and by how 
much they are uncertain. 

This digression on parametric modeling errors and the presentation of the SWLQR results that follows 
is vital. To achieve high levels of performance, which means using high-gain control, LQR requires an 
accurate plant model. Ifa model contains parametric modeling errors and high-gain LQR control (p — 0) 
is used, then the closed-loop poles will go to cancel zeros of G(s) which will be in different locations than 
the true zeros. Such errors often lead to instability, and this is why we worry about parametric modeling 
errors. 


17.4.3.2 SWLQR 


Designing feedback controllers that are robust to parametric uncertainty is a very difficult problem. 
Research on this problem is ongoing, and a great deal of research has gone into this problem in the past, 
see [4] for a survey on this topic. Here we present only one of the many methods, SWLQR, that can be 
used to design LQR controllers that are robust to parametric modeling errors. While the treatment is 
by no means complete, it should give one a sense of how to deal with parametric modeling errors when 
designing LQR controllers. 

The philosophy behind SWLQR is that it aims, as the title implies, to desensitize the LQR controller to 
parametric uncertainty. The advantage of this philosophy is that it produces a technique that is nothing 
more than a special way to choose the weighting matrices of the cross-weighted LQR problem presented 
in Section 17.4.1. A shortcoming of the method is that it does not result in a controller that is guaranteed 
to be robust to the parametric uncertainty in the model. 

In presenting the SWLQR results, the vector of uncertain parameters to which you wish to desensitize 
a standard LQR controller will be denoted by a, and a; will denote the ith element of a. To desensitize 
LQR controllers to the uncertain variables in a, a quadratic penalty on the sensitivity of the state to each 
of the uncertain variables «;, 


Ox! (t) Ox(t) 
Oa; nae Oa; 


(17.27) 


is added to the cross-weighted LQR cost functional (Equation 17.22). In this expression ox is known as 


the sensitivity state of the system and Roa; is the sensitivity state weighting matrix, which must be positive 
semidefinite. Quadratic penalties for each a; (Equation 17.27) are included in the LQR cost functional to 
tell the mathematics that we care not only about deviations in the state from its equilibrium position but 
that we also care about how sensitive the value of the state is to the known uncertain parameters of the 
system. The larger the value of Ryq;, the more we tell the cost that we are uncertain about the influence of 
parameter a; in our model on the trajectory of the state. In turn, the larger the uncertainty in the influence 
of parameter a; on the state trajectory, the more robust the resulting SWLQR controller will be to the 
uncertain parameter 4. 

Making certain assumptions about the dynamics of the sensitivity states leads to the SWLQR result [6,7]. 
It is not necessary to understand or discuss these assumptions to design SWLQR controllers, and we thus 
simply present the SWLQR result. 


The SWLQR Result 


Given a stable open-loop system (Equation 17.21), let « denote the vector of uncertain variables to which 
you want to desensitize a LQR controller and define the sensitivity state weighting matrices Rag, such that 


T 
Rao; = Rog, 20 where Raa; € R'*” 


17-14 Control System Advanced Methods 


Then the SWLQR controller results from synthesizing the cross-coupled cost LQR controller from 
Theorem 17.2 with the following set of design weights 


n 
i! “AAT + _, 0A 
Ry = Rex + » Das Raul Aa (17.28) 
nN 
OAT oo Bris): 
Rew = Rew + d at Roa,A a (17.29) 
nN 
, OR 2 _, OB 
Ris Rode Dy Bayt Raw AT! aS (17.30) 


In these expressions ng denotes the number of uncertain parameters in a, and Rex, Rey and R,,, are the 
nominal values of the weights from a standard, unrobust LQR design. The matrix derivatives 0A/Ou; and 
OB/00; are taken term by term. For example, 0A/Oa; is a matrix of the same size as A whose elements are 
the partial derivatives with respect to a; of the corresponding elements in the A matrix. For 


Ay Aiz -:+ Ain 
Al 


Ant Ann 
OA, OA. OAin 
0a; Oa; Oa; 


OA21 
OA al Wea 


OAnl OAnn 
00; 00; 


While it might not be obvious that the modified cost expressions (Equations 17.28 through 17.30) lead to 
controllers that are more robust to the uncertain parameters in a, extensive empirical and experimental 
results verify that they do [7]. To see this mathematically, consider the state cost, x! (t)Ryxx(t) from 
Equation 17.28, when the variables we wish to keep small are defined by z(t) = Cx(t), 


aAt OA 
x? (t)Rayx(t) = x7 (t)C? Cx(t) + x? (t) —A "Rug, A! —x(t) 
Oa, Oay 
aAt dA 
e+e +x? (f()—A "Rog, A! —x(t) 
Ong & On, 
This expression can also be expressed as z! (t)z(t) with z(t) = Cx(t) where C? = [c! Cr tee Cina! 
and Cy, is the minimal order matrix square root of 
AT A 
AT RaAm ae 
Oa; Oa; 


From this we see that the SWLQR cost modifications essentially add new variables to the vector of variables 
we wish to keep small, z(t). In addition to the standard variables we wish to keep small for performance 
defined by Cx(t), we tell the cost to keep the variables Cy;x(t) small, and these are related to the sensitivity 
of the state to parametrically uncertain variables. Furthermore with this new z(t) vector, the zeros of G,(s) 
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will change. As a result, in the high-gain limit as p — 0 the closed-loop poles will not go to cancel the 
uncertain zeros defined by the original vector of variables we wish to keep small, z(t) = Cx(t). 

Selecting the values of the state sensitivity weights Raq, is, as to be expected, an iterative process. 
Typically Row; is chosen to be either B,J or BiRex where 8; is a positive scalar constant. Both these choices 
simplify the selection of Rua; by reducing the choice to a scalar variable. When using Ryo; = BiRxx, one 
tends to directly trade off performance, reflected by the nominal state cost x! (t)Ryxx(t) with robustness 
to the parametric error now captured by the state cost 


OAT yes aA 
Bix" (t)=—A*RyA7! ——x(t) 
Ou; 00; 
On the other hand, selecting Rug; = BiJ is a way of telling the cost that you care about both performance 
and robustness since the various terms in the state cost are not directly related as when Rog; = Bi Rex. 


Example 17.2: 


Consider the mass, spring, dashpot system shown in Figure 17.6. The four masses are coupled 
together by dampers with identical coefficients, c; = 0.05 Vi, and springs with identical stiffnesses, 
kj =1Vi. In this system, the mass m3 is uncertain, but known to lie within the interval 0.25 < 
m3 < 1.75 while the values of m;, m2, and mg are all known to be 1. A unit intensity, zero-mean, 
white noise disturbance force &(t) acts on mq, and the control u(t) is a force exerted on m2. The 
performance variable of interest is the position of the tip mass m4, so z(t) = qa(t). The design goals 
are to synthesize a controller which is robust to the uncertainty in m3 and that rejects the effect 
of the disturbance source on the position of m4. A combination of SWLQR and the stochastic LOR 
results from Section 17.4.2 will be used to meet these design goals. 
Using Newton’s laws, the dynamics of this system can be expressed as 


M(t) + C(t) + Ka(t) = Ty, u(t) + T2&(t) (17.31) 
where q! (t) = [qi(t) qz(t) qa(t) qa(t)]’, 
m 0 0 0 0 0 
_|0 m0 0 ella i 50 
PAS i 5 5G m3 0 = 6), 12= 0 
0 0 0 m 0 1 
ky +k2 —kz 0 0 cy +02 -C 0 0 
K= —kz kp + k3 —k3 0 C= -C 2+ —C3 0 
a 0 —k3 k3 +k, —kg = 0 -C3 34+, -—C4 
0 0 —k, kg 0 0 —C4 C4 
Then by defining the state vector as 
q(t) 
t=]. 17.32 
x(t) Hal ( ) 
1 qy(t) 1 Got) 1 qz(t) '  qa(t) 
' a(t) = qalé) 


FIGURE 17.6 Mass, spring, dashpot system from Example 17.2. 
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the dynamics from Equation 17.31 can be represented in the state space as 
x(t) = Ax(t) + Bu(t) + Lé(t) 
with 


0 | 0 0 
Bear | Fem eget 
With the states defined by Equation 17.32, the performance variables we wish to keep small are 


z(t)=Cx(t) whereC=[0 0 0 1 0 0 0 Qj (17.33) 


The first step in designing a SWLQR controller is to determine satisfactory values for the nominal 
design weights Ryx, Rw, and R,,. This can be done by ignoring the uncertainty and designing a LQR 
controller that achieves the desired system performance. Given the single-input, single-output (SISO) 
system with the desire to control the position of m4, we find a nominal LQR controller by minimizing the 
stochastic LQR cost functional 


j=E| im i. . [z7 et) + pult)?] ar} o>0 (17.34) 
ot Jo 


Substituting z(t) from Equation 17.33 into the cost brings the integrand into the form of that in the 
cross-weighted cost from Equation 17.22 with Rex = C™C, Ryy = 0, and R,,, = p. Since m3 is uncertain, 
we let it take on its median value of m3 = 1 to synthesize actual controllers. After some iteration over the 
value of p, we decide that p = .01 leads to a nominal LQR controller that achieves good performance. 

Figure 17.7 shows a set of transient responses for the mass spring system to an impulse applied to the 
disturbance source of the system. Responses for various values of the uncertain mass m3 are shown, to 
illustrate how well the controllers behave in the presence of the parametric uncertainty. Notice that the 
LQR controller designed assuming m3 = 1 is not stable over the entire range of possible values for m3 
and that even when the LQR controller is stable for values of m3 # 1 the performance degrades. A better 
way to view the performance robustness of this controller to the uncertain mass in the presence of a white 
noise disturbance source is shown in Figure 17.8. This figure compares the root mean square (RMS) value 
of the tip mass position, q4(t), normalized by its open-loop RMS value as a function of the value of the 
uncertain mass m3. Such plots are often called cost “buckets,” for obvious reasons. When the closed-loop 
system is unstable, the cost blows up so that the wider the cost “bucket” the more robust the system will 
be to the parametric uncertainty. The cost bucket of Figure 17.8 clearly shows that the LQR controller 
is not robust to the parametric modeling error in m3, since the cost “bucket” completely lies within the 
bounds on m3. 

Now to robustify this nominal LQR controller, we use the SWLQR modified costs from Equations 17.28 
through 17.30. Here a = m3, 28 = 0 and 


om — 
aA 0 0 
a =| _ ame) a(M7!) 
Om3 Om3 K oma Cc 
where 
0 0 0 0 
aM _|0 0 0 0 
am; «| 0 0 —m;* 0 
0 0 0 0 


Using Rug = BI and m3 = 1, we found that B = 0.01 led to a design that was significantly more robust 
than the LQR design. This can be seen in Figures 17.7 and 17.8 that compare the transient response 
and cost “buckets” of the SWLQR controller to the LQR controller. Both figures demonstrate that the 
SWLAQR controller is stable over the entire range of possible values for m3. As to be expected though, we 
do sacrifice some nominal performance to achieve the stability robustness. 
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FIGURE 17.7 Transient response of the 4 mass, spring, dashpot system to an impulse applied at the disturbance 
source &(t) for various values of m3. The open-loop transients are shown with dots, the LQR transients are shown 
with a dashed line, and the SWLQR transients are shown with a solid line. 


17.4.4 LQR with Frequency Weighted Cost Functionals 


As discussed in Section 17.2, there is often a gap between what a LQR controller achieves and the desired 
control system performance. This gap can essentially be viewed as the discrepancy between the time 
domain optimization method, LQR, and the control system specifications. Often, both control system 
performance and robustness specifications can be expressed and analyzed in the frequency domain, but 
the time domain nature of the LQR problem makes it difficult to take advantage of these attributes. Using 
frequency weighted cost functionals to synthesize LQR controllers leads to a useful and practical variation 
on the standard LQR problem that narrows the gap between the time and frequency domains. 

Recall that the quadratic cost functional is our means of communicating to the mathematics the desired 
control system performance. We’ve discussed the physical interpretation of the cost in the time domain, 
and now to understand it in the frequency domain we apply Parseval’s Theorem to it. 
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FIGURE 17.8 Comparison of cost “buckets” between the LQR and SWLQR controllers. 


Theorem 17.3: Parseval’s 


For a vector valued signal h(t) defined on —oo < t < +00 with 


is h! (t)h(t) dt < 00 


(oe) 


if we denote the Fourier transform of h(t) by h(go) then 


a h' (t)h(t) dt = -. ie h* (gm) h(go) dw 
ser 27 Jago 


where h*(ja) = h™ (—jw) is the complex conjugate transpose of h(ja). 


Applying this theorem to the quadratic cost functional from Equation 17.3 with R = pI and perfor- 


mance variables z(t) = Cx(t) produces the cost 


i= = / [2*(jw)z(go) + pur (ge) u(7o)] doo 


—0Co 


Notice that the time domain weight, p, remains constant in the frequency domain. This illustrates that 
the standard LQR cost functional tells the mathematics to weight the control equally at all frequencies. 
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To influence the desired control system performance on a frequency by frequency basis, we employ 
the frequency weighted cost functional 


1 Co 

jax / [2* Qo) WF (900) Wi (0)z(g0) + U* Go) WS (Joo) Wa (gen) u(g0)] do (17.35) 
—0O 

in which both Wj (jw) and W2(jw) are user-specified frequency weights [8]. Looking at the cost (Equa- 

tion 17.35) for a SISO system in which the design weights will be scalar functions produces 


oo 
1= =f lmGePeGo)? +1020)? 4G0)2] de 
2: fn) 

and helps to illustrate the benefit of using frequency weighted cost functionals. The frequency weights 
allow us to place distinct penalties on the state and control cost at various frequencies, which is not possible 
to do when constant weights are used. Their main advantage is that they facilitate the incorporation of 
known and desired control system behavior into the synthesis process. This is why frequency weighting 
plays a key role in modern control design [9]. If one knows over what frequency range it is important 
to achieve performance and where the control energy must be small, this knowledge can be reflected 
into the frequency weights. Since we seek to minimize the quadratic cost (Equation 17.35), large terms 
in the integrand incur greater penalties than small terms and more effort is exerted to make them small. 
For example, if there is a rigorous bandwidth constraint set by a region of high-frequency unmodeled 
dynamics and if the control weight W2(jw) is chosen to have a large magnitude over this region then 
the resulting controller would not exert substantial energy in the region of unmodeled dynamics. This in 
turn would limit the controller’s bandwidth. Similar arguments can be used to select W (jo) to tell the 
controller synthesis at what frequencies the size of the performance variables need to be small. 

To synthesize LQR controllers that minimize the frequency weighted cost functional (Equation 17.35), 
it is necessary to transform the cost functional back into the time domain. Doing so requires state space 
realization of the frequency weights Wj(jw) and W2(ym). Hence we let 


z1(s) = Wi (s)z(s) with Wi(s) = Cy(sl — Ay)7!B, + D, (17.36) 
Z2(s) = W2(s)u(s) with Wo(s) = Co(sI —A>)7!By + Dy (17.37) 
Using these definitions and Parseval’s Theorem, the cost (Equation 17.35) becomes 


je A i7AGalnge+euane)|de= i 


QTd 65 —0o 


[zr (t)z(t) +22 (Na(0)| dt (17.38) 


This cost can be brought into the form of an LQR problem that we know how to solve by augmenting 
the dynamics of the weights (Equations 17.36 and 17.37) to the open-loop dynamics (Equations 17.1 
and 17.2). Using x;(t) and x(t) to denote, respectively, the states of W(s) and W2(s) to carry out the 
augmentation produces the system 


X(t) = AX(t) + Bu(t) (17.39) 
Z(t) =CX(t) + Du(t) 
with 
A 0 0 B : ‘ 
AAS We Os|) BEV gale a | p=(5| 
0 0 <A Bo a 


where ¥7 (t) = [x7 (t) x) x} (t)] and Z! (t) = [zi (t) zi (t)] contains the outputs of W1(s) and W2(s). 
Notice that 


2" (Z(t) =z (t)zi(t) +z} (Hzr(t) 
= XT (t)CTCX(t) + 247 (t)C’ Du(t) + ul (YD Dut) 
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and thus the frequency weighted cost (Equation 17.38) becomes 
CO 
J= / [eT eTcx(t) +247 (CT Dut) + u" (YD Du(s) | dt 
—o0o 


Realize that minimizing this cost subject to the dynamic constraints (Equation 17.39) is not only equivalent 
to minimizing the frequency weighted cost functional (Equation 17.35), but it is also a simple manner to 
carry out the optimization by using the cross-weighted LQR results from Theorem 17.2 with 


Rv=C CO Rye R= DD (17.40) 
Applying Theorem 17.2 produces the control law 


x(t) 
u(t) = —KX(t) = —[Ky K, Ko] | xi(t) (17.41) 
x2(t) 


where K = R,,} (RZ, + B'S), S is the solution to the Riccati equation 
s(A- BR, Ri) +(AT- RuRia B") S+ (Re Rak) — SBR;1B7S =0, 


and the control weights are defined by Equation 17.40. 

Clearly the augmented state vector X(t) must be fed back to implement the control law (Equa- 
tion 17.41). Since x;(t) and x(t) are the fictitious states of the weights W1(s) and W2(s), it is necessary 
to deliberately create them in real time to apply the feedback control law (Equation 17.41). That is, using 
frequency weighted cost functionals to synthesize LQR controllers leads to dynamic compensators. The 
control law is no longer a set of static gains multiplying the measurement of the state vector x(t). It is a 
combination of the dynamics of the weighting matrices and the static gains K. A block diagram show- 
ing the feedback architecture for the dynamic LQR controller arrived at by minimizing the frequency 
weighted cost functional of Equation 17.35 is shown in Figure 17.9. Realize that having to implement a 
dynamic compensator is the price we pay for using frequency weights to incorporate known frequency 
domain information into the LQR controller synthesis. Though since the order of the compensator is 
only equal to the combined order of the design weights, this fact should not be troublesome. 
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FIGURE 17.9 Block diagram for the LQR controller synthesized using the frequency weighted cost from 
Equation 17.35. 
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17.4.4.1 Selecting the Frequency Weights 


Although we do not present the results here, it is possible to use any combination of frequency weights 
and constant weights when synthesizing LQR controllers with frequency weighted costs. For example, 
one could use a constant state penalty, W1(s) =J, with a frequency weighted control. In any case, to 
satisfy the underlying LQR assumptions one must use stable design weights and ensure that D} D2 > 0 
(so that Ry exists). 

When synthesizing LQR controllers with frequency weighted cost functionals, the iteration over the 
values of the design weights involves selecting the dynamics of multivariable systems. Although it is not 
necessary, choosing the weighting matrices to be scalar functions multiplying the identity matrix, 


Wi(s)=wyi(s)I and W(s) = w2(s)I 


simplifies the process of selecting useful weights. Then the process of selecting the weights reduces to 
selecting the transfer functions w)(s) and w2(s) so that their magnitudes have the desired effect on the cost. 

To arrive at a useful set of weights, one must use one’s knowledge of the physics of the system and 
the desired control system performance. Keeping this in mind, simply select the magnitude of w2(s) to 
be large relative to |w2(0)| over the frequency range where you want the control energy to be small and 
choose wj(s) to have a large magnitude relative to |w (0)| over the frequency regions where you want 
the performance variables to be small. It is important to note that here large is relative to the values of 
the DC gains of the weights, |w(0)| and |w2(0)|. The DC gains specify the nominal penalties on the state 
and control cost in a manner similar to the constant weights of a standard LQR problem. As such, when 
choosing the frequency weights, it is beneficial to break the process up into two steps. The first step is to 
choose an appropriate DC gain to influence the overall characteristics of the controller, and the second 
step is to choose the dynamics so that the magnitudes of the weights reflect the relative importance of the 
variables over the various frequency ranges. 

When choosing the frequency weights it is vital to keep in mind that the controls will have to be large 
over the frequency ranges where the performance variables are to be small. Likewise, if the control energy 
is specified to be small over a frequency range, it will not be possible to make the performance variables 
small there. Even though we could tell the cost to make both the performance variables and the control 
signals small in the same frequency range through the choice of the frequency weights, doing so will most 
likely result in a meaningless controller since we are asking the mathematical optimization problem to 
defy the underlying physics. 


17.5 Mini-Max and H,, Full-State Feedback Control 


Consider the problem of rejecting the effect of the disturbances, d(t), on the performance variables of 
interest, z(t), for the system 


x(t) = Ax(t) + Bu(t) + Ld(t) (17.42) 
z(t) = Cx(t) 


LQR control is not well suited to handle this problem because the optimal control that minimizes the 
quadratic cost (Equation 17.3) subject to the dynamic constraints (Equation 17.42) wants to know the 
future values of the disturbances, which is not realistic. The stochastic version of the LQR problem is also 
inappropriate unless d(t) is white noise, which is rarely the case. To deal optimally with the disturbances 
using a full-state feedback controller, it is necessary to adopt a different philosophy than that of the LQR. 
Rather than treating the disturbances as known or white noise signals, they are assumed to behave in a 
“worst case” fashion. Treating the disturbances in this way leads to the so called Ho full-state feedback 
controller. Ho controllers have become as popular as LQR controllers in recent years as a result of their 


17-22 Control System Advanced Methods 


own attractive properties [10]. We introduce H.o controllers here using the quadratic cost functional 
optimization point of view. 

The “worst case” philosophy for dealing with the disturbances arises by including them in the quadratic 
cost functional with their own weight, y, much in the same way that the controls are included in the LQR 
cost functional. That is we seek to optimize the quadratic cost 


J(u, d) = ; i is ExGzo + pu! (t)u(t) - yd" (Hao)] dt y,p>0 (17.43) 
0 


subject to the dynamic constraints of Equation 17.42. In this optimization problem both the controls and 
disturbances are the unknown quantities that the cost is optimized over. Note that since the disturbances 
enter the cost functional as a negative quadratic, they will seek to maximize J(u,d). At the same time 
the controls seek to minimize J(u, d) since they enter the cost functional as a positive quadratic. Hence, 
by using the cost (Equation 17.43), we are playing a mini-max differential game in which nature tries 
to maximize the cost through the choice of the disturbances, and we as control system designers seek to 
minimize the cost through the choice of the control u(t). This mini-max optimization problem can be 
compactly stated as 

min max J(u, d). (17.44) 


Since nature is allowed to pick the disturbances d(t) which maximize the cost, this optimization 
problem deals with the disturbances by producing a control law that is capable of rejecting specific worst 
case disturbances. 

The solution to the mini-max optimization problem (Equation 17.44) is not guaranteed to exist for all 
values of y. When the solution does exist, it produces a full-state feedback control law similar in structure 
to the LQR controller. The solution to the mini-max differential game is summarized in the following 
theorem. 


Theorem 17.4: Mini-Max Differential Game 


Given the system dynamics 
x(t) = Ax(t) + Bu(t) + Ld(t) (17.45) 


with x(t) € R", u(t) € R™, and d(t) € R? along with the performance variables we wish to keep small 
z(t) = Cx(t) with z(t) € R?, we define the mini-max quadratic cost functional 


1 e,0) 
Hund) => [ [e"@a()+ ou" (Qut)—vraT(nae] at y,0>0 
0 
in which p and y are user-specified design variables that weight the relative influence of the controls and 


disturbances. Under the following assumptions 


1. The entire state vector x(t) is available for feedback 
2. d(t) is a deterministic, bounded energy signal with [5° d'(t)d(t) dt < co 
3. Both[A B] and [A L] are stabilizable, and [A C] is detectable 


If the optimum value of the cost J(u, d) constrained by the system dynamics (Equation 17.45) exists, it is a 
unique saddle point of J(u, d) where 


1. The optimal mini-max control law is 


1 
u(t) = —Kx(t) with K = —B'S (17.46) 
p 
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2. The optimal, worst case, disturbance is 
lor 
d(t)= GL’ Sx(t) 
Y 
3. Sis the unique, symmetric, positive semidefinite solution of the matrix Riccati equation 
T . ry ee eee 
SA+A°S+C°C—S| —BB’ — SLL’ )S=0 (17.47) 
p Y 


4. The closed-loop dynamics for Equation 17.45 using Equation 17.46 
X(t) = (A — BK)x(t) + Ld(t) (17.48) 
are guaranteed to be asymptotically stable. 


If the solution to this optimization problem exists, it produces a stabilizing full-state feedback controller 
with the same structure as the LQR controller but with a different scheme for evaluating the feedback 
gains. Since the L matrix of Equation 17.45 appears in the Riccati equation 17.47, the mini-max control 
law directly incorporates the information of how the disturbances impact the system dynamics. The facts 
that the min-max controller guarantees a stable closed-loop and takes into consideration the nature of 
the disturbances make it an attractive alternative to LQR for synthesizing controllers. 


17.5.1 Synthesizing Mini-Max Controllers 


Mini-max controllers are not guaranteed to exist for arbitrary values of the design weight y in the quadratic 
cost functional J(u, d). Since y influences the size of the — 41 term in the Riccati equation 17.47, there 
will exist values of y > 0 for which there is either no solution to the Riccati equation or for which S will 
not be positive semidefinite. It turns out that there is a minimum value of y, ymin, for which the mini-max 
optimization problem has a solution. Hence, useful values of y will lie in the interval ymin < y < 00. Note 
that as y — oo, the Riccati equation 17.47 becomes identical to the LQR one from Equation 17.5, and we 
recover the LQR controller. Likewise when y = ymin, we have another special case which is known as the 
full-state feedback Ho controller*. For any other value of y in ymin < y < 00 we still have an admissible 
stabilizing mini-max controller. 

As with LQR, synthesizing mini-max controllers requires solving an algebraic Riccati equation. How- 
ever, the presence of the — LL" term in the mini-max Riccati equation 17.47 makes the process of 


computing an S = S? > 0 that satisfies Equation 17.47 more complicated than finding an S = S? > 0 that 
satisfies the LQR Riccati equation 17.5. The reasons for this are directly related to the issue of whether or 
not a solution to the mini-max optimization problem exists. 

To understand how one computes mini-max controllers, it is necessary to understand how current 
algebraic Riccati equation solvers work. While there is a rich theory for the topic, we summarize the key 
results in the following theorem. 


Theorem 17.5: The Algebraic Riccati Equation 


The Riccati equation 
A™S+SA+SVS—Q=0 (17.49) 


* See Chapter 18 for details. 
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is solved by carrying out a spectral factorization™ of its associate Hamiltonian matrix 


A V 
‘es E Je 


IfV= —BB',Q=—-C!C,[A_ Blisstabilizable,and[A  C] is detectable then the Riccati equation solvers 
produce the unique, symmetric, positive semidefinite solution, S = S' > 0, to the Riccati equation 17.49. 
Otherwise, as long as H has no jw-axis eigenvalues, the spectral factorization can be performed and a 
solution, S, which satisfies Equation 17.49 is produced. 


The Hamiltonian matrix for the mini-max Riccati equation, Hy, is 


A LiL? — “BB 
Hy= ¥ (17.50) 
=CC —At 


The sign indefinite nature of the 1/y?LL’ — 1/pBB" term in Hy makes it quite difficult to numerically test 
whether or not a solution to the mini-max optimization problem exists. Thus, a constructive algorithm 
based on the existing algebraic Riccati equation solvers is used to synthesize mini-max controllers. 


Theorem 17.6: Algorithm for Computing Mini-Max Controllers 


Pick a value of y and check to see if Hy from Equation 17.50 has any jw-axis eigenvalues. If it does, 
increase y and start over. If it does not, use an algebraic Riccati equation solver to produce a solution S 
to Equation 17.47. Test if S > 0. If it is not, increase gamma and start over. If S > 0, check to see if the 
closed-loop dynamics from Equation 17.48 are stable. If they are not, increase gamma and start over. If they 
are, you have constructed a mini-max controller, since you’ve found a S = S! > 0 that satisfies the Riccati 
equation 17.47. 


Theoretically, the final step of the algorithm, which requires checking the closed-loop stability is not 
necessary. However, we highly recommend it since the numerical stability of the solution to the Riccati 
equations for values of y near Ymin can be questionable. 

From the algorithm for computing mini-max controllers, it can be seen that evaluating Ho. controllers 
for a fixed value of p will require a trial-and-error search over y. To compute ymin it is best to use a 
bisection search over y in the algorithm to find the smallest value of y for which S = S’ > 0 and the 
closed-loop system (Equation 17.48) is stable. Current control system design packages such as MATLAB 
and MATRIXx employ such algorithms for computing the H. controller that in turn determines ymin. 

While the mini-max and H.. controllers are quite distinct from LQR controllers all the advice given 
for designing LQR controllers applies to mini-max controllers as well. Namely, it is necessary to iterate 
over the values of the design weights and independently check the robustness and performance char- 
acteristics for each design when synthesizing mini-max and Ho controllers. It can be shown through 
manipulating the Riccati equation 17.47 that the robustness properties of LQR controllers from Sec- 
tion 17.3 apply to Hoo and mini-max controllers as well. Furthermore, modifications of the powerful 
sensitivity and frequency weighted LQR design tools from Section 17.4 do exist and can be used to incor- 
porate stability robustness and known design specifications into the controller synthesis of mini-max and 
Hoo controllers. 


* Spectral factorizations are essentially eigenvalue decompositions. 
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18.1 Introduction 


The fundamentals of output feedback H2 (linear quadratic Gaussian or LQG) and Ho controllers, which 
are the primary synthesis tools available for linear time-invariant systems, are presented in an analogous 
and tutorial fashion without rigorous mathematics. Since Hz and Hoo syntheses are carried out in the 
modern control design paradigm, a review of the paradigm is presented, along with the definitions of the 
Hz and Hoo norms and the methods used to compute them. The state-space formulae for the optimal 
controllers, under less restrictive assumptions than are usually found in the literature, are provided in an 
analogous fashion to emphasize the similarities between them. Rather than emphasizing the derivation 
of the controllers, we elaborate on the physical interpretation of the results and how one uses frequency 
weights to design Hoo and Hz controllers. Finally, a simple disturbance rejection design for the longi- 
tudinal motion of an aircraft is provided to illustrate the similarities and differences between Hoo and 
Hz controller synthesis. 


18.2 The Modern Paradigm 


Hz and Hoo syntheses are carried out in the modern control paradigm. In this paradigm both performance 
and robustness specifications can be incorporated in a common framework along with the controller 
synthesis. In the modern paradigm, all of the information about a system is cast into the generalized 
block diagram shown in Figure 18.1 [1-3]. The generalized plant, P, which is assumed to be linear and 
time-invariant throughout this article contains all the information a designer would like to incorporate 
into the synthesis of the controller, K. System dynamics, models of the uncertainty in the system’s 
dynamics, frequency weights to influence the controller synthesis, actuator dynamics, sensor dynamics, 
and implementation hardware dynamics from amplifiers, and analog-to-digital and digital-to-analog 
converters are all included in P. The inputs and outputs of P are, in general, vector valued signals. The 
sensor measurements that are used by the feedback controller are denoted y, and the inputs generated 
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K <—_____ 


FIGURE 18.1 Generalized block diagram of the modern paradigm. 


by the controller are denoted u. The components of w are all the exogenous inputs to the system. 
Typically these consist of disturbances, sensor noise, reference commands, and fictitious signals that 
drive frequency weights and models of the uncertainty in the dynamics of the system. The components 
of z are all the variables we wish to control. These include the performance variables of interest, tracking 
errors between reference signals and plant outputs, and the actuator signals which cannot be arbitrarily 
large and fast. 

The general control problem in this framework is to synthesize a controller that will keep the size of 
the performance variables, z, small in the presence of the exogenous signals, w. For a classical disturbance 
rejection problem, z would contain the performance variables we wish to keep small in the presence of the 
disturbances contained in w that would tend to drive z away from zero. Hence, the disturbance rejection 
performance would depend on the “size” of the closed-loop transfer function from w to z, which we shall 
denote as Tzy(s). This is also true for a command following control problem in which z would contain 
the tracking error that we would like to keep small in the presence of the commands in w that drive the 
tracking error away from zero. 

Clearly then, the “size” of T,,(s) influences the effect that the exogenous signals in w have on z. Thus, 
in this framework, we seek controllers that minimize the “size” of the closed-loop transfer function T;,,(s). 
Given that T;,,(s) is a transfer function matrix, it is necessary to use appropriate norms to quantify its size. 
The two most common and physically meaningful norms that are used to classify the “size” of Tz(s) are 
the Hz and Ho norms. As such, we seek controllers that minimize either the Hz or Hog norm of Tzy(s) 
in the modern control paradigm. 


18.2.1 System Norms 


Here we define and discuss the Hz and Hoo norms of the linear, time-invariant, stable system with transfer 
function matrix 


G(s) = C(sl — A) 1B 


This notation is meant to be general, and the reader should not think of G(s) as only the actuator 
to sensor transfer function of a system. Realize that G(s) is a system and thus requires an appro- 
priate norm to classify its size. By a norm, we mean a positive, scalar number that is a measure 
of the size of G(s) over all points in the complex s-plane. This is quite different from, for exam- 
ple, the maximum singular value of a matrix, Omax[A], which is a norm that classifies the size of the 
matrix A. 
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The H2 Norm 


Definition 18.1: 71. Norm 
The Hz norm of G(s), denoted ||G||2, is defined as 


i 


a ha 2 1 [ex 2 
|Gll2 = (-/ trace [G(gw)G* (ya) | aw) = (3, [= Deteooe) 


—0CO 


where 0; denotes the ith singular value, G*(jw) is the complex conjugate transpose of G(jw), and r is the 
rank of G(jw). 


The Hz norm has an attractive, physically meaningful interpretation. If we consider G(s) to be the 
transfer function matrix of a system driven by independent, zero mean, unit intensity white noise, u, then 
the sum of the variances of the outputs y is exactly the square of the Hz norm of G(s). That is 


Ely" yO] = IGOI (18.1) 


The 2 norm of G(s) thus gives a precise measure of the “power” or signal strength of the output of a 
system driven with unit intensity white noise. Note that in the scalar case \/E[y! (t)y(t)] is the RMS or 
root mean squared value for y(t) so the Hz norm specifies the RMS value of y(t). A well-known fact 
for stochastic systems is that the mean squared value of the outputs can be computed by solving the 
appropriate Lyapunov equation [4]. As such, a state space procedure for computing the Hz norm of G(s) 
is as follows [2]. 


Computing the Hz Norm 
If L, denotes the controllability Gramian of (A, B) and Ly the observability Gramian of (A, C), then 


AL. +L-A' +BB! =0 A'L,+L,A+C'C=0 


and , 
z 


1 
|Gll2 = [trace(ct.c7)| _ [trace(B LB) | 


Note that this procedure for computing the H2 norm involves the solution of linear Lyapunov equations 
and can be done without iteration. 


The H.. Norm 


Definition 18.2: 71., Norm 


The Hoo norm of G(s), denoted ||G|loo, is defined as 


|Glloo = SUP OmaxlG)] 


In this definition “sup” denotes the supremum or least upper bound of the function omax[G(yw)], and 
thus the Ho norm of G(s) is nothing more than the maximum value of Omax[G(jw)] over all frequencies 
w. The supremum must be used in the definition since, strictly speaking, the maximum of Omax[G(y)] 
may not exist even though omax[G(jw)] is bounded from above. 
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Hoo norms also have a physically meaningful interpretation when considering the system y(s) = 
G(s)u(s). Recall that when the system is driven with a unit magnitude sinusoidal input at a specific 
frequency, Omax[G(jo)] is the largest possible output size for the corresponding sinusoidal output. Thus, 
the Hoo norm is the largest possible amplification over all frequencies of a unit sinusoidal input. That is, 
it classifies the greatest increase in energy that can occur between the input and output of a given system. 
A state space procedure for calculating the Ho) norm is as follows. 


Computing the Hoo Norm 
Let ||Glloo = Ymin- For the transfer function G(s) = C(sI] — A)~!B with A stable and y > 0, ||Glloo <y if 


and only if the Hamiltonian matrix 
A - ppt 
H= Y 
=c'c. =A? 


has no eigenvalues on the jw-axis. This fact lets us compute a bound, y, on ||G||oo such that ||Glloo < y. 
So to find ymin, select ay > 0 and test if H has eigenvalues on the jw-axis. If it does, increase y. If it does 
not, decrease y and recompute the eigenvalues of H. Continue until ymin is calculated to within the desired 
tolerance. 

The iterative computation of the Ho. norm, which can be carried out efficiently using a bisection search 
over y, is to be expected given that by definition we must search for the largest value of Omax[G(yw)] over 
all frequencies. 

Note, the Hz norm is not an induced norm, whereas the H. norm is. Thus, the Hz norm does not 
obey the submultiplicative property of induced norms. That is, the Hoo norm satisfies 


G1 Galloo ¥ IGrllooll Galloo 


but the 72 norm does not have the analogous property. This fact makes synthesizing controllers that 
minimize || Tz(s)|loo attractive when one is interested in directly shaping loops to satisfy norm bounded 
robustness tests*. On the other hand, given the aforementioned properties of the 12 norm, synthe- 
sizing controllers that minimize ||Tzy(s)||2 is attractive when the disturbances, w, are stochastic in 
nature. In fact, Hz controllers are nothing more than linear quadratic Gaussian (LQG) controllers 
so the vast amount of insight into the well-understood LQG problem can be readily applied to H2 
synthesis. 


Example 18.1: 


In this example the Hz and Hoo norms are calculated for the simple four-spring, four-mass system 
shown in Figure 18.2. The equations of motion for this system can be found in Example 17 2 in 
Chapter 17. The system has force inputs on the second and fourth masses along with two sensors that 
provide a measure of the displacement of these masses. The singular values of the transfer function 
from the inputs to outputs, which we denote by G(s), are shown in Figure 18.3. The Ho norm of 
the system is equal to the peak of 0; = 260.4, and the Hz norm of the system is equal to the square 
root of the sum of the areas under the square of each of the singular values, 14.5. Note that when 
considering the H2 norm, observing the log log plot of the transfer function can be very deceiving, 
since the integral is of oj, not log(o;), over w, not log w. 


As pointed out in the example, the differences between the Hoo and Hz norms for a system G(s) are best 
viewed in the frequency domain from a plot of the singular values of G(yw). Specifically, the Hop norm is 
the peak value of Omax[G(yw)] while the H2 norm is related to the area underneath the singular values of 
G(ym). For a more in-depth treatment of these norms the reader is referred to [1,2,5,6]. 


* See Chapter 20 for a detailed exposition of this concept. 
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FIGURE 18.2 Mass, spring, dashpot system from Example 18.1. For the example kj = mj; = 1 Vi, and cj = 0.05 Vi. 


18.3 Output Feedback 1, and H2 Controllers 


Given that all the information a designer would like to include in the controller synthesis is incorporated 
into the system P, the synthesis of Hz and H. controllers is quite straightforward. In this respect all of 
the design effort is focused on defining P. Below, we discuss how to define P using frequency weights to 
meet typical control system specifications. Here we simply present the formulas for the controllers. 

All the formulas will be based on the following state-space realization of P, 
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FIGURE 18.3 Singular values of the transfer function between the inputs and outputs of the mass, spring system 


shown in Figure 18.2. 
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This notation is a shorthand representation for the system of equations 


x(t) = Ax(t) + By w(t) + Bou(t) (18.2) 
z(t) = C,x(t) + Dy, w(t) + Dj2u(t) (18.3) 
y(t) = Cox(t) + Dz w(t) + D22u(t) (18.4) 


Additionally, the following assumptions concerning the allowable values for the elements of P are made. 


Assumptions on P 


1. Dy =0 (A.1) 
2. [A Bp] is stabilizable (A.2) 
3. [A  C,] is detectable (A.3) 
By Vox Vy 
4, V= BE pi): *1>0 with Vy >0 AA 
Fal 1 1 ik Vel with Vyy > (A.4) 
Cr Re Re 
5. R=| Lf, Dp) := >0 withR,, >0 (A.5) 
Ee Re, Ruy me 


Assumption A.1 ensures that none of the disturbances feed through to the performance variables which 
is necessary for Hz synthesis but may be removed for Hoo synthesis (see [7] for details.) Assumptions A.2 
and A.3 are needed to guarantee the existence of a stabilizing controller while the remaining assumptions 
are needed to guarantee the existence of positive semidefinite solutions to the Riccati equations associated 
with the optimal controllers. 


Theorem 18.1: 712 Output Feedback 


Assuming that w(t) is a unit intensity white noise signal, E[w(t)w! (t)] = 18(t — 1), the unique, stabilizing, 
optimal controller which minimizes the Hz norm of Tz(s) is 


A+ BoF,+L L2D22F, | —L 
Res + BF, +12C,+1,DyFy | —La (18.5) 
F) | 0 
where 
Py =—Ri} (Ri, + By X2) 
In=—(%2CF + Vy) Vy! (18.6) 
and Xz and Y2 are the unique, positive semidefinite solutions to the following Riccati equations 
- T —lpT —1 pT 
OSA A OP Ry RR IRS BRB (18.7) 
0 =AeY2 + YoAZ + Vix — VayVyy Vay — Y2Cz Vy," CoYo (18.8) 


where 


Ar = (A = Boi} Ri, and Ac = (A tg Vey Vig C2) 
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Theorem 18.2: H.. Output Feedback [8] 


Assuming that w(t) is a bounded L2 signal, [°. w" (t)w(t) dt < 00, a stabilizing controller which satisfies 
I Tzw@)\loo < y is 


A | —Zis Ling 
Kee 18. 
00 | Foo | 0 (18.9) 
where 
Ago = At (Bi + LopD21) Woo + Bo Foo + ZooLoo C2 + ZooLooD22Foo 
where 


2 1 
Foo = Riga (Ri, + BY Xoo) Woo = Bl Xoe 


iT -1 
i —(YooC? re Viy) V5) Zoo = (1 = v2 YooKs:] 


and Xoo and Yo are the solutions to the following Riccati equations 


_ 2 1 

OSA A Ke PR RRR ee (B2Rt = “iB ) Xo (18.10) 
2 e 1 

0 = Ac Yoo + YooAg + Vex — VayVyy' Vay — Yoo (c} Vy C2 — = cfc) vs (18.11) 


that satisfy the following conditions 


1. Xo > 0 
2. The Hamiltonian matrix for Equation 18.10, 


A-B)R,}R1, = —B R71 BE + 57 Bi By 


_ = T 
Rex + Ruki Re,  —(A—BRz}R2,) 


u uu ~ xu 


has no jo)-axis eigenvalues, or equivalently A + By Woo + BoFoo is stable 
cae eat 
The Hamiltonian matrix for Equation 18.11, 


T 
(A-VoVy!Q)  -CEVy!a+Sclo 


-lyT —1 
= Vax + Vey Vy Vey —A+ Vey Vy C2 


has no jo)-axis eigenvalues, or equivalently A + LogC2 + Z Yeo ct C, is stable 


5. 0(YooXoo) < y?, where p(-) = max; |;(-)| is the spectral radius 


The (sub)optimal central Hoo controller which minimizes ||Tzw||oo to within the desired tolerance is Koo 
with y equal to the smallest value of y > 0 that satisfies conditions 1 to 5. 


Unlike the H2 controller, the Ho» controller presented here is not truly optimal. Since there is no 
closed-form, state-space solution to the problem of minimizing the infinity norm of a multiple-input, 
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multiple-output (MIMO) transfer function matrix Tz(s), the connections between the mini-max opti- 
mization problem 


inf sup : * [z" ete) zs vw" Qw(0)| dt (18.12) 
uw Jo 


and Hoo optimization are used to arrive at the constructive approach for synthesizing suboptimal 
Hoo controllers given in Theorem 18.2. In fact, satisfying the conditions 1 to 5 of Theorem 18.2 is 
analogous to finding a saddle point of the optimization problem (Equation 18.12), and the search for 
Ymin is analogous to finding the global minimum over all the possible saddle points. As such, any value 
of y > Ymin Will also satisfy conditions 1 to 5 of Theorem 18.2, and thus produce a stabilizing controller. 
Such controllers are neither Hz nor Hoo optimal. Since in the limit as y > oo the equations from The- 
orem 18.2 reduce to the equations for the H2 optimal controller, controllers with values of y between 
Ymin and infinity provide a trade off between Hoo and H2 performance. Along these lines, it is also worth 
noting that there is a rich theory for mixed Hz /Hoo controllers that minimize the Hz norm of Tzy(s) 
subject to additional Ho constraints. See [9-11] for details. 

The value of w(t) that maximizes the cost in Equation 18.12 is known as the worst case disturbance, 
as it seeks to maximize the detrimental effect the disturbances have on the system. In this regard, 
Hoo controllers provide optimal disturbance rejection to worst case disturbance, whereas the H2 
controllers provide optimal disturbance rejection to stochastic disturbances. 

Both Hz and H. controllers are observer-based compensators [2], which can be seen from their block 
diagrams, shown in Figures 18.4 and 18.5. The regulator gains F) and Fo arise from synthesizing the 
full-state feedback controller, which minimizes the appropriate size of z! (t)z(t) constrained by the system 
dynamics Equation 18.2. Then the control law is formed by applying these regulator gains to an estimate 
of the states x(t). The states, x(t), are estimated using the noisy measurements of y(t) from Equation 18.4, 
and Lz and ZooLoo are the corresponding filter gains of the estimators. 

In particular, F> is the full-state feedback LQR gain that minimizes the quadratic cost 


jic=t {im : i i [720 ar| 


constrained by the dynamics of Equation 18.2, and L» is the Kalman filter gain from estimating the states 
x based on the measurements y(t). Under the assumption that z(t) is an ergodic process* 


Tug = lim : / “eM ne(nat = [2 wet] = (Tew (18.13) 
TOO 0 


and this is exactly why H2 synthesis is nothing more than LQG control. 


By k 
y ye & + u 
- + 
> Ly rO— >| f 2 Hh > 
+ +K+ 
vy v 
Ak 
+Kt C.K 
Dy 


FIGURE 18.4 Block diagram of Kz from Equation 18.9. Note, the Kalman Filter estimate of the states x(t) from 
Equation 18.2, x2(t), are the states of K. 


* Assuming z(t) is ergodic implies that its mean can be computed from the time average of a measurement of z(t) as 


t—> oo [4]. 
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FIGURE 18.5 Block diagram of K.. from Equation 18.9. Note, the Hoo optimal estimate of the states x(t) from 
Equation 18.2, Xoo(t), are the states of Koo, and w(t) is an estimate of the worst case disturbance. 


Analogously, Foo is the full-state feedback Hoo control gain that results from optimizing the mini-max 
cost of Equation 18.12, and Woo is the full-state feedback gain that produces the worst case distur- 
bance which maximizes the cost of Equation 18.12*. Unlike the Kalman filter in the 12 controller, the 
Hoo optimal estimator must estimate the states of P in the presence of the worst case disturbance which 
is evident from the block diagram of Koo shown in Figure 18.5 [12]. This is why the filter gain of the 
Hoo optimal estimator, ZooLo9, is coupled to the regulator portion of the problem through Xoo from 
Equation 18.10. 

Since the Hz controller is an LQG controller, the closed-loop poles of Tz(s) separate into the closed- 
loop poles of the regulator, eig(A — Bz F2), and estimator, eig(A — L2C2). A consequence of this separation 
property is that the Hz Riccati equations (Equations 18.7 and 18.8) can be solved directly without iteration. 
Since the worst case disturbance must be taken into consideration when synthesizing the Hoo optimal 
estimator, the regulator and estimator problems in the Ho synthesis are coupled. Thus, the Ho controller 
does not have a separation structure that is analogous to that of the H2 controller. In addition, the 
Hoo Riccati equations (Equations 18.10 and 18.11) are further coupled through the y parameter, and we 
must iterate over the value of y to find solutions of the Ho Riccati equations that satisfy conditions 1 to 
5 of Theorem 18.2. 

Note that in the literature the following set of additional, simplifying assumptions on the values of 
the elements of P are often made to arrive at less complicated sets of equations for the optimal Hoo and 
Hp, controllers [6,13,14]. 


Additional Assumptions on P 


1. Dy =0 (No control feed-through term) 

2. CT Dy =0 (No cross penalty on control and state) 

3, B \Di ,=0 (Uncorrelated process and sensor noise) 

4, DID, 2=1 (Unity penalty on every control) 

D: DI jPa1 =1 (Unit intensity sensor noise on every measurement) 


* See the section on Hoo Full State Feedback in Chapter 17 for details. 
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18.4 Designing Hz and H. Controllers 


The results presented in the previous section are powerful because they provide the control system 
designer with a systematic means of designing controllers for systems whose entire state cannot be fed 
back. In order to take full advantage of these powerful tools, it is up to the designer to communicate 
to the optimization problems the desired control system performance and robustness. In the modern 
paradigm, this is done through the choice of the system matrix P. Since systems and their associated 
desired performance are diverse, there is no systematic procedure for defining P. However, by exploiting 
the rich mathematics of the Hz and Hoo optimization problems along with their physical interpretations, 
it is possible to formulate guidelines for selecting appropriate systems P for a wide variety of problems. 

Regardless of the synthesis employed, P will contain both the system model and the design weights used 
to communicate to the optimization the desired control system performance. Any linear interconnection 
of design weights and model can be selected so long as Assumptions A.1 through A.5 are satisfied. To 
satisfy the assumption that Ry > 0, all of the control signals must appear explicitly in z. This is to be 
expected, since we cannot allow the synthesis to produce arbitrarily large control signals. Similarly, to 
ensure Vy, > 0, every measurement y must be corrupted by some sensor noise so as to avoid singular 
estimation problems. 

Frequency-dependent weighting matrices are often included in P, since they provide greater freedom 
in telling the synthesis the desired control system performance. The synthesis of 2 and Hoo controllers 
with frequency weights is just as straightforward as classical LQG synthesis with constant weights. Once 
the interconnection of the model with the defined performance variables, disturbances, and weights is 
specified, it is just a simple matter of state augmentation and block diagram manipulation to realize the 
state space form of P in Equations 18.2 through 18.4. Then given a state space representation of P, the 
formulas for the optimal controllers found in Theorems 18.1 and 18.2 can be used. The ability to use any 
admissible system interconnection with any combination of frequency weights is a direct consequence of 
the fact that we build an estimator for the entire state of P into the controllers. As such, the dynamics of 
any frequency weights will be reflected in the compensator whose order will be the same as that of P. 

In either the Hz or Hoo framework, arriving at a satisfactory design will involve iteration over the 
values of the frequency weights. Thus, it is vital to have an in-depth understanding of the dynamics of the 
system when choosing the system interconnection and the values for the design variables. 


18.4.1 H2 Design 


Given that the H2 optimal controller is an LQG controller, it is useful to adopt a stochastic framework and 
use the insights afforded by the well-known LQG problem when selecting P for Hz synthesis, see [13,15]. 
In this respect, w(t) must contain both the process and sensor noises, while z(t) must contain linear 
combinations of both the states and controls. Furthermore, the system P must be comprised of the system 
model and all the design weights such as the noise intensities and the state and control weighting matrices. 
For example, Figure 18.6 illustrates a possible system interconnection for the classical LQG problem of 
minimizing a weighted sum of state and control penalties given a system whose dynamics 


x(t) = Ax(t) + Bu(t) + Ld(t) (18.14) 
y(t) = Cx(t) + v(t) 


are driven by the uncorrelated stochastic disturbances, d(t), and sensor noise, v(t). 

In the interconnection of Figure 18.6, W; are weighting matrices, or design variables, that the designer 
selects. For the classical LQG problem, all the W; are constant matrices. Since w! (t)= [wh (t) wh (t)] 
must be a unit intensity, white noise process, W; and W) are the matrix square roots of the intensity 
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FIGURE 18.6 Block diagram interconnection for a typical P(s). 


matrices for the process and sensor noises d and v such that 


d(t ww 0 
e[E]ero veo] P the? 


As for the performance variable weights, W3 is a weight on the outputs that produces a particular state 
weighting, and W4 is the matrix square root of the control weighting matrix. These define the cost Jaq 
from Equation 18.13 to be 


iose lim : / ‘ [s7@cTws W3Cx(t) + ul QW] Wau) ar} 
T=? 00: 0 


A drawback of classical LQG synthesis is that the weighting matrices are constant and thus limit our 
ability to place distinct penalties on the disturbances and performance variables at various frequencies. 
When synthesizing H2 controllers the weights W; can, in general, be functions of frequency. Since per- 
formance and robustness specifications are readily visualized in the frequency domain, using frequency 
weights provides much more freedom in telling the optimization problem the desired control system 
behavior. 

When choosing the values of the frequency weights, one should use the fact that H2 synthesis is 
equivalent to LQG synthesis. Any frequency weights that appear on the performance variables can be 
chosen using the insights afforded by the LQR problem with frequency weighted cost functionals as a result 
of Equation 18.13%. In brief, one uses Parseval’s Theorem to arrive at a frequency domain representation 
of JLq from Equation 18.13. For the system interconnection shown in Figure 18.6 with scalar frequency 
weights W3(s) = w3(s)J and Wa(s) = wa(s)I 


1 CO 
a sf [wslow)!yf(s0)yn (40) + wae)" Go)ue)] de 


From this expression of the 12 cost, it is clear that the weights should be chosen to have a large magnitude 
over the frequencies where we want the outputs, y,, and controls, u, to be small. 

Frequency weights that appear on the disturbance signals should be viewed as shaping filters that 
specify the spectral content of the process and sensor noises. The values of the weights can then be chosen 


* Section 4.4 in Chapter 17 has a detailed exposition of this. 
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to capture the true spectral content of the disturbances, as w(t) must be unit intensity white noise to apply 
Theorem 18.2, or they can be chosen to influence the controller to produce some desired behavior. For 
example, in the system shown in Figure 18.6 if W;(s) = w;(s)J, then the control will work hard to reject 
the disturbances, d, over the frequencies where |w1(jo)| is large. Likewise, if W2(s) = W2(s)I and |w2(ya)| 
is large over a particular frequency range, then the controller will not exert much effort there because we 
are telling the synthesis that the sensor measurements are very noisy there. 


18.4.2 H. Design 


In the Hoo framework it is possible to use loop shaping, see [1], to achieve performance and robustness 
specifications that can be expressed in the frequency domain. This is due to the fact that 


ITewlloo <¥ > II (Tew); lloo <¥ Vii (18.15) 


where (Tew) i denotes the closed-loop transfer function matrix between exogenous disturbance w; and 
performance variable zj. To take advantage of Equation 18.15, it is necessary to define P so that the 
closed-loop transfer function matrices we wish to shape appear directly in T,,,(s) and are multiplied by 
frequency-dependent design weights. 

These concepts are best illustrated through an example. Consider the system Equation 18.14, which 
can be represented in the frequency domain as 


y(s) = Gy(s)d(s) + Go(s)u(s) + v(s) 
G1(s) = C(sI — A)“!L 
G)(s) = C(sI — A)~!B 


where the disturbances are now considered to be unknown but bounded £2 signals. Suppose that we are 
interested in designing a controller that rejects the effect of the disturbances d(t) on the outputs y(t) and 
that is robust to an unstructured additive error in the input to output system model. Then it is necessary 
to independently shape the closed-loop transfer function between d and y, S(s)G;(s), and the closed-loop 
transfer function K(s)S(s). In particular, we require S(s)Gj(s) to have a desirable shape, and we need to 
satisfy the standard additive error stability robustness test 


1 
OmaxLK (7)S(g@)] << ——— 
lea(go)| 
where S(s) = [I — Go(s)K (s)]7! and e,(s) is a transfer function whose magnitude bounds the additive 
error *. If W; =I and W2 =I, then the system interconnection shown in Figure 18.6 is suitable for 
designing Ho» controllers that achieve the loop-shaping objectives, because 


W3S(s)Gy(s) W3C(s) 


WaK(s)S(s)Gi(s) WaK(s)S(s) rea) 


Tzw(s) = 


where C(s) = S(s)G2(s)K(s). 

Notice that both of the loops of interest, S(s)G(s) and K(s)S(s), appear directly in Equation 18.16 
multiplied by the design weights. By selecting scalar frequency-dependent weights, W3 = w3(s)I and 
Wg = wa(s)I, an Hoo controller that achieves a specific value of y ensures that 


OmaxlS(7@)Gi(yo)] < a ae (18.17) 
|w3(7@)| 

OmaxLK (yw) S(yo)] < ae (18.18) 
|w4(7)| 


as a result of Equation 18.15. Similar bounds will also hold for the other (Tew) in Equation 18.16. To 
take advantage of Equations 18.17 and 18.18, set y = 1 and select the values of w3(s) and w4(s) to provide 


* See Chapter 9. 
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desirable bounds on S(s)G;(s) and K(s)S(s). For example, let w4(s) = ea(s). Then if the H. controller 
based on these values of the weights achieves ||T'z||oo © 1, the desired loops will in fact be shaped to 
satisfy Equations 18.17 and 18.18. This is how one should choose the values of the design variables to 
shape the loops of interest in an Hoo design. 

In using this method of weight selection there are a few issues the designer must keep in mind. First 
of all, realize that the bounds implied by Equation 18.15 and exemplified by Equation 18.17 are not 
necessarily tight over all frequencies. As a result it helps to graphically inspect all the constraints implicit 
in the choice of T,,,(s) as one iterates through the values of the design variables. More importantly, simply 
assuming y = 1 when the values of the weights are chosen does not ensure an Hoo controller that achieves 
| Tewlloo © 1. In fact, when ||Tzy|loo >> 1 it is a strong indication that the values of the design variables 
impose unrealistic constraints on the system’s dynamics. One cannot choose w3(s) and wa(s) arbitrarily. 
They must complement each other. Another reason why the design variables cannot be chosen arbitrarily 
involves the fact that || (Tew) lloo < y Vi,j. Not only will w3(s) shape the weighted sensitivity transfer 
function S(s)G,(s), it will also shape C(s). Since S(s) + C(s) = I, there will clearly be restrictions on the 
choice of w3(s). While loops such as C(s) may not be of primary interest, they will influence the overall 
performance of the controller and should be kept in mind when selecting the values of the weights. 

The choice of P in Figure 18.6 with W; = W2 = I could also have been made using structured singular 
value concepts*. In this context, the performance variables z, and disturbances w2 can be viewed as the 
inputs and outputs to an unknown but norm bounded unstructured uncertainty that captures the additive 
error in the input to output model. Likewise, the performance variables z; and disturbances w; can be 
viewed as the inputs and outputs to a fictitious, unknown, norm bounded unstructured uncertainty that 
captures the desire to reject the disturbances d at the outputs y,. Then selecting the values of the design 
weights is akin to scaling the system in the same way that the D-scales, used in the D-K iteration, scale 
the system. 


18.4.3 Additional Notes for Selecting Design Weights 


To ensure that Assumptions A.1 through A.5 are satisfied once the dynamics of the frequency weights 
are augmented to the system model, it is necessary to use proper, stable, minimum phase weights. For 
example, in the system shown in Figure 18.6, W4(s) must contain an output feedthrough term to ensure 
Ruu > 0. 

An important issue to be aware of when using frequency weights is that it is possible to define a set 
of weights with repetitive information. For example, in the system of Figure 18.6 with W2(s) = wa(s)I 
and W4(s) = wa(s)I, specifying the magnitudes of w2(s) and wa(s) to be large over the same frequency 
region tells both optimization problems the same information, make the controls small there. Not only 
is such information redundant, it is also undesirable, since the order for the compensator is equal to the 
order of P. 


18.5 Aircraft Design Example 


To illustrate more clearly how one uses frequency weights to design Hz and H. controllers, we shall 
discuss the design of a wind gust disturbance rejection controller for a linearized model of an F-8 aircraft. 
As you shall see, the modern paradigm allows us to incorporate frequency domain performance and 
robustness specifications naturally and directly into the controller synthesis. 

The F-8 is an “old-fashioned” aircraft that has been used by NASA as part of their digital fly-by-wire 
research program. Assuming that the aircraft is flying at a constant altitude in equilibrium flight allows 
us to linearize the nonlinear equations of motion. In doing so, the longitudinal dynamics decouple from 


* See Chapter 20 for more details. 
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the lateral dynamics. The variables needed to characterize the longitudinal motion, which are defined 
in the schematic drawing of the F-8 shown in Figure 18.7, are the horizontal velocity, v(t), pitch angle, 
Q(t), pitch rate, q(t) = A(t), angle of attack, a(t), and flight path angle, B(t) = 0(t) — a(t). To control the 
longitudinal motion, elevators, §,(t), and flaperons, 3, (t), which are just like the elevators except that they 
move in the same direction, were used. While the thrust also influences the longitudinal motion of the 
aircraft, it is considered to be constant in our designs. The measurements are the pitch and flight path 
angles, y(t) =[0(f) B(t)]. 

The effect of wind gust disturbances, which primarily corrupt the angle of attack, is modeled as the 
output of a shaping filter driven with unit intensity white noise, d(t). Using the state vector 


x"(t)=[0(t) Bt) g(t) v(t) xa(t)] 


in which xq(t) is the state of the first-order shaping filter of the wind gust disturbance model, the linearized, 
longitudinal equations of the F-8 aircraft are 


x(t) = Ax(t) + Bu(t) + Ld(t) (18.19) 
y(t) = Cx(t) + v(t) 


with u! (t) = [8,(t) 3¢(é)] and 


0.0 0.0 1.0 0.0 0.0 0.0 0.0 
150 —1.50 0.0 0.0057 1.50 0.160 0.80 
A=| -120 120 —060 —0.0344 -120| B=| —199 —3.0 
—0.8520 0.290 0.0 —0.0140 —0.290 —0.0115 ——0.0087 
0.0 0.0 0.0 0.0 —0.730 a nef 


100 0 
L'=[0.0 0.0 0.0 0.0 1.1459] c=| 1 


0 1 0 0 0 


The units for the angles and control signals are in degrees while the velocity has units of ft/s. The outputs 
are modeled as a nominal signal with additive white noise v(t) that has an intensity of 1 = 0.01 deg”/s, 
E {v7 (t)v(t)} = L18(t — t), to capture the limited accuracy of the sensors. 

The objective is to design controllers that reduce the effect of the wind disturbance on the system. 
Specifically we would like the magnitude of each output to be less than 0.25 degrees up to 1.0 rad/s as 
the aircraft passes through wind gusts. In addition, we require the control system to be robust to an 
unstructured multiplicative error reflected to the output of the plant whose magnitude is bounded by the 
function 


€m(S) = 0.15" 


0=71 
Bee ke Horizontal | 


B= 9 
Velocity, v 


d(t)- Wind gust disturbance 


FIGURE 18.7 Definition of variables for the longitudinal dynamics of the F-8. 
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This multiplicative error captures the unmodeled dynamics associated with the flexibility of the aircraft’s 
airframe. It will essentially constrain the bandwidth of the design to prevent these unmodeled modes 
from being excited. 

Both of the design specifications can be represented in the frequency domain. To meet the performance 
specification, we require the closed-loop transfer function from d to y, S(s)G;(s), to satisfy 


OmaxlS(o)G1(yw)] < 0.25 for0<w < 1.0 rad/s (18.20) 
where 


S(s) = [I — Gy(s)K(s)]" 
G)(s) = C(sI — A)~!B 
G(s) = C(sI — A) !L 


To ensure stability robustness to the multiplicative error we require that 


OmaxlC(ym)] < (18.21) 


|em(I)| 
where C(s) = S(s)G2(s)K(s). 

Given this representation of the design goals, we shall synthesize Hz and H. controllers that shape 
the closed-loop transfer functions S(s)G;(s) and C(s) to meet these constraints. Using the system inter- 
connection for P shown in Figure 18.8 makes good mathematical and physical sense for this problem. 
Mathematically, P(s) shown in Figure 18.8 leads to the following closed-loop transfer function matrix, 


(18.22) 


des wi)S)Gi)_——_/fBWals)C(S) 


pw (s)K(s)S(s)Gi(s) — P./W2(s)K(s)S(s) 


Notice that the loops of interest, S(s)G;(s) and C(s), appear directly in Equation 18.22 and are directly 
influenced by the scalar frequency weights w1(s) and wa(s). Realize that the coloring filter dynamics for 
the wind gust disturbance are already included in the system dynamics Equation (18.19), so that w1(s) 
should not be viewed as a shaping filter for d. Rather wj(s) is a design variable that overemphasizes the 
frequency range in which the impact of d is most vital, and it is chosen to reflect in the optimization 
problem our desire to appropriately shape S(s)G,(s). The scalar constant weight p, which is a penalty on 
the control that must be included in the synthesis to satisfy Assumption A.5, was allowed to vary, whereas 
yw was held fixed to capture the limited sensor accuracy. P(s) also makes good sense in terms of the physics 
of the design objectives. It includes the effects of both the process and sensor noises, and its performance 
variables, z, contain the outputs we wish to keep small in the presence of the disturbances. 

To illustrate that there is a strong connection between the physical, stochastic motivation used to select 
the values of the weights in the 12 framework and the more mathematical norm bound motivation used 
in the Hoo framework, we compare the results of an Hoo and an H2 design that both use the same values 
of the weights. The weights were chosen as described in the previous section. After some iteration we 
found that with 

o0=0.01, wy(s)= PSE and w2(s)= SOOUeste2) 
(s+ 1.25) 3.5(s + 1000) 
both the Ho and H2 designs met the desired performance and robustness specifications. Note that as a 
result of using these frequency weights, the controllers had eight states. 

Figures 18.9 and 18.10 show that the loops of interest have in fact been shaped in accordance with the 
design goals. As seen in Figure 18.9, which compares the open and closed-loop disturbance to output 
transfer functions, both designs meet the performance goal from Equation 18.20. In this figure, 1/|w1(j)| 
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FIGURE 18.8 Generalized system P(s) used in the F-8 designs. 
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FIGURE 18.9 Comparison of the open- and closed-loop disturbance to output transfer functions for the Hoo and 
Hz designs. The frequency weight W(s) used to shape S(s)Gj(s) is also shown. 


is also shown to illustrate how the value of w;(s) was chosen. From an H2 perspective, the |w1(jo)| is 
large over 0 < w < 1.0 rad/s and small elsewhere to tell the synthesis that the intensity of the disturbance 
is large where we desire good disturbance rejection. In the context of the Hoo design, we assumed that 
y = land selected 1/|w (jo)| to appropriately bound omax[S(y)G(yw)] in accordance with the fact that 


OmaxlS(9@)Gi(yw)] < aad, Yi (18.23) 


|wi(y@)| 
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FIGURE 18.10 Comparison of omax[C(yw)] for the Hoo and Hz designs. The bound on the multiplicative error, 
1/\em(y)|, and the frequency weight W2(s) used to shape C(s) are also shown. 


for any Hoo design which achieves || Tz ||oo < y. For the values of the weights given above, || Tz ||oo = 0.95 
for the Hoo design, which ensures that Equation 18.23 was satisfied. A difference in the performance 
achieved by both designs is expected since the same values of the weights are used to minimize different 
measures of the size of Tz,(s). 

Figure 18.10, which compares Omax[C(yw)] for the designs, illustrates that the stability robustness 
criterion from Equation 18.21 is also satisfied. Again, 1/,/{t|w2(j@)| is shown to illustrate the manner in 
which the value of w2(s) was chosen. As seen in the figure, w2(s) forces the Hoo design to roll off below 
the stability robustness bound, 1/|e(jo)|, in accordance with the fact that 


Omax[C(yw)] < Vo. (18.24) 


a 
JE |W2(9@)| 


Since |W2(ya)| is large beyond 5.0 rad/s, it tells the Hz synthesis that the sensor noise is large there, which 
in turn limits the control energy beyond 5.0 rad/s. 

The value of ¢ also played an important role in the designs. In the H2 design, adjusting p directly 
influenced the amount of control effort used, just as a control weight would in LQG synthesis. For the 
Hoo design, p minimized the constraints that the values of w;(s) and wo(s) placed on the closed-loop 
transfer functions K(s)S(s)G;(s) and K(s)S(s) in Equation 18.22. Since these loops are not of primary 
interest, choosing a small value of p ensured that wj(s) and w2(s) would not overly constrain these since, 
for example, 


Y 


OmaxLK (yw) S(7@)G1 (j@)] Sas 
plwi(7o)| 


for any Hoo design. 
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The similarities in the achieved loop shapes are not coincidental. In fact, the dynamics of the Hoo and 
Hz controllers presented here are quite similar. There is a clear reason why the similarity exists even 
though the optimization problems used are distinct. Once all of the desired control system performance 
is incorporated into P(s) via the design variables, the task of minimizing the Hz norm of T,,,(s) becomes 
nearly identical to the task of minimizing the Ho. norm of Tz (s). This can be seen in Figure 18.11, 
which compares the values of Omax[Pzw(yw)] and Omax[Tzw(y@)] for the two designs. Here P,,,(s) denotes 
the open-loop transfer function matrix between w and z of P(s). As such, Omax[Pzy(jo)] is an indication 
of the nominal cost that the controllers seek to minimize. Specifically, to minimize the Ho norm of 
Tzw(s) the peak in Omax[Pzy(y)] must be flattened out so that it looks like a low pass filter. Then the 
DC gain of the filter must be reduced to further minimize the Hoo norm of T;,,(s). This is also the case 
for minimizing the H2 norm of Tzy(s) which is dominated by the area under the spike in Omax[Pzw(yw)] 
(recall that the area is evaluated linearly and that we are using a log log plot). While the optimization 
problems are distinct, the manner in which the cost is minimized is similar. 

Figure 18.11 also provides a clear indication of how the optimization problems differ. Notice that the 
Hz design rolls off faster than the Hoo design. This is because the 12 design minimizes energy, or the 
area under Omax[Tzw(ya)], at the expense of its peak value, whereas the Ho design seeks to minimize the 
peak of Omax[Pzw(yw)] at the expense of allowing there to be more energy at higher frequencies. 

It would be improper to draw conclusions about which synthesis approach is better based on these 
designs, especially since the same values for the weights in P(s) were used. Rather, our intent has been to 
illustrate the connections between the Ho and H2 frameworks and how one can go about synthesizing 
Hoo and Hz controllers to meet frequency domain design specifications. We should also note that the 
methodology used in this example has been applied to and experimentally verified on a much more 
complex system [16]. 


Maximum singular values of T,,,(s) and P,,,(s) 
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FIGURE 18.11 Comparison of the open-loop “cost,” Omax[Pzw(jw)], and the closed-loop “cost,” Omax[ Tzw(g@)], for 
the Hoy and H2 controllers. 
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19.1 Introduction* 


Feedback controllers are designed to achieve certain performance specifications in the presence of both 
plant and signal uncertainty. Typical controller design formulations consider quadratic cost functions 
on the errors primarily for mathematical convenience. However, practical situations may dictate other 
kinds of measures. In particular, the peak values of error signals are often more appropriate for describing 
desired performance. This can be a consequence of uniform tracking problems, saturation constraints, 
rate limits, or simply a disturbance rejection problem. In addition, disturbance and noise are in general 
bounded and persistent because they continue to act on the system as long as the system is in operation. 
Such signals are better described by giving information about both the signals’ frequency content and 
time-domain bounds on peak values. 

The above class of problems motivated a formulation that involves the Peak-to-Peak gain of a system, 
which is mathematically given by the ¢; norm of the pulse response. This formulation was first reported 
in [13], and the problem was completely solved in several subsequent articles [5-8,11]. The extension 
of the theory to incorporate plant uncertainty was reported in [3,4,9,10]. An extensive coverage of this 
theory with detailed references can be found in [2]. 

The need for developing this problem was further intensified by the failure of the frequency-domain 
techniques to address time-domain specifications. For instance, attempting to achieve an overshoot 


* Research Supported by AFOSR under grant AFOSR-91-0368, by NSF under grant 9157306-ECS, and by Draper Laboratory 
under grant DL-H-467128. 
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constraint using Hoo or H2 by appropriately adjusting the weighting matrices can be a very frustrating 
experience. On the other hand, solutions to such problems will no longer be in closed form due to 
the complexity of the performance objectives. Exact or approximate solutions will be obtained from 
solving equivalent yet simpler optimization problems. The derivation of such simpler problems and the 
computational properties are essential components of this research direction. 


19.1.1 A Design Tool 


The motivation behind research in ¢; theory is developing a design tool for MIMO uncertain systems. A 
powerful design tool in general should have three ingredients: 


1. Practicality: The ability to translate a large set of performance specifications into conditions or 
constraints readily acceptable by the design tool. It is evident that not all design specifications can 
be immediately translated into mathematical conditions. However, the mathematical formulation 
should well approximate these objectives. 

2. Computability: It is in general straightforward to formulate controller design problems as con- 
strained optimization problems. What is not so straightforward is formulating problems that can 
be solved efficiently and with acceptable complexity. 

3. Flexibility: The ability to alter a design to achieve additional performance specifications with small 
marginal cost. 

It is evident that practicality and computability are conflicting ingredients. Computational com- 
plexity grows as a function of several parameters, which include the dimension of the system, the 
uncertainty description, and the performance specifications. Flexibility makes it possible to design 
a controller in stages, i.e., by altering a nominally good design to achieve additional specifications. 


19.1.2 Motivation 


We give some reasons behind the development of such a design tool, by quoting from the book: Control 
of Uncertain Systems: A Linear Programming Approach. 


1. Complex systems: Many of today’s systems, ranging from space structures to high purity distillation 
columns, are quite complex. The complexity comes from the very high order of the system as well 
as the large number of inputs and outputs. Modeling such systems accurately is a difficult task and 
may not be possible. A powerful methodology that deals systematically with multiple inputs and 
outputs and with various classes of structured uncertainty is essential. 

2. High performance requirement: Systems are built to perform specific jobs with high accuracy. 
Robots are already used to perform accurate jobs such as placing components on integrated circuit 
boards. Aircraft are built with high maneuverability and are designed specifically for such tasks. 
Classical SISO design techniques cannot accommodate these problems resulting in designs that 
are conservative and perform poorly. 

3. Limits of performance: In complex systems, it is time-consuming to establish, by trial and error, 
whether a system can meet certain performance objectives (even without uncertainty). Thus, it is 
necessary to develop systematic methods to quantify the fundamental limitations of systems and 
to highlight the trade-offs of a given design. 

4. A systematic design process: It is inevitable that designing a controller for a system will involve 
iterations. Unless this procedure is made systematic, the design process can become very cumber- 
some and slow. The design procedure should not only exhibit a controller. It should also provide 
the designer with indicators for the next iteration, by showing which of the constraints are the 
limiting ones and which part of the uncertainty is causing the most difficulty. Note also that a 
general procedure should be able to accommodate a variety of constraints, both in the time and in 
the frequency domain. 
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5. Computable methods: It is quite straightforward to formulate important control problems, but it 
is not so easy to formulate solvable problems that can provide computable methods for analysis 
and synthesis. Much of the current research invokes high level mathematics to provide simple 
computable results. The computability of a methodology is the test of its success. By computability 
we do not mean necessarily closed form solutions. Problems that give rise to such solutions 
have limited applicability. However, computability means that we can synthesize controllers via 
computable algorithms and simultaneously obtain qualitative information about the solution. 

6. Technological advancement: Many aspects of technological development will affect the design of 
control systems. The availability of “cheaper” sensors and actuators that are well-modeled allows 
for designing control systems with a large number of inputs and outputs. The availability of fast 
microprocessors, as well as memory, makes it possible to implement more complex feedback 
systems with high order. The limiting factor in controller implementation is no longer the order 
of the controller. Instead, it is the computational power and the memory availability. 

7. Available methods: The available design techniques have concentrated on frequency-domain per- 
formance specifications by considering errors in terms of energy rather than peak values. These 
methods (such as H2 and Ho.) have elegant solutions. However, this elegance is lost if additional 
time-domain specifications (e.g., overshoot, undershoot, settling time ..) are added. This created 
a need for a time-domain based computational methodology that can accommodate additional 
frequency-domain constraints. This design tool aims at achieving this objective. 


In the sequel we will summarize the ¢; design tool by discussing the three ingredients, practicality, 
computability, and flexibility. 


19.2 Practicality 
19.2.1 Examples 


To motivate the formulation of the ; problem, we begin with two examples. 

The first is the control design for an Earth Observing System (EOS). EOS is a spacecraft that orbits the 
earth and points in a specific location. It carries on its platform various sensory instruments, with the 
objective of collecting data from earth. An example of such an instrument is an array of cameras intended 
to provide images of various points and landscapes on earth. The spacecraft is subjected to various kinds 
of disturbances: external pressures, noise generated from the instruments on board, and the spacecraft 
itself. The objective of the control design is to point the spacecraft accurately in a specific direction, 
otherwise known as attitude control. 

The second example is the control design of an active suspension of an automobile. A simplified 
one-dimensional problem is shown in Figure 19.1. 

The objective of the controller design is to maximize ride comfort, while simultaneously maintaining 
handling (road holding ability) in the presence of bad road conditions. 

These examples have several common features: 


1. The objectives in both problems are to keep the maximum deviations of signals from set points 
bounded by some prescribed value to attain uniform tracking or disturbance rejection. In the EOS 
example, performance is measured in terms of maximum deviations of the attitude angles from a 
set point. In the active suspension problem, performance is measured in terms of the maximum 
force acting on the system and the maximum deviation of the distance of the wheel from the 
ground. 

In mathematical terms, both performance specifications are stated in terms of peak values of 
signals, ie., 


IlWlloo = max sup |w(t)]. 
i=l,...nm ¢ 
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FIGURE 19.1 Active suspension. 


This is known as the £.9-norm of the signal (or its peak value). For the active suspension problem, 
the performance can be stated as 
Cr Le 
is Co 


for some performance bound y. It is straightforward to incorporate additional scalings and weights 
into the performance. 

2. The disturbances in both examples are unknown, persistent in time, but, bounded in magnitude 
(peak value). A good model for the disturbance in both cases is given by 


d=Ww,  |lwllo <1, 


where W is a linear time-invariant filter that gives information about the frequency content of 
the disturbance. This model of disturbance accommodates persistent disturbances that are not 
necessarily periodic. It does not assume that the signal dies out asymptotically. 

3. In both problems, saturation constraints are quite important and play a role in limiting the per- 
formance. In the active suspension problem, the saturation constraint is given by the maximum 
deflection of the hydraulic actuator, i.e., 


lest — Xalloo S Ysat- 


These constraints combined with the performance objectives have to be satisfied for all distur- 
bances w such that || w]loo < 1. 

4. Both examples are difficult to model precisely, and thus the control strategies have to accommodate 
unmodelled dynamics. In this article, we will not discuss in detail the robust performance problems. 
We refer the reader to [2] for details. 
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It is evident from the above discussion that peak values of signals are natural quantities in stating design 
specifications. 


19.2.2 Formulation 


Figure 19.2 shows a general setup for posing performance specifications. The variables as defined in the 
figure are 


u = control inputs 
y = measured outputs 


fixed commands 
unknown commands 
disturbances 


w = exogenous Inputs = 
noise 


tracking Errors 
control Inputs 
measured Outputs 
states 


z = regulated outputs = 


The operator G is a 2 x 2 block matrix mapping the inputs w and u to the outputs z and y: 


bile: ol 

y Go, Goa} Lu’ 

The actual process or plant is the submatrix G22. Both the exogenous inputs and the regulated outputs are 

auxiliary signals that need not be part of the closed loop system. The feedback controller is denoted by K. 
From our discussion above, the set of exogenous inputs consists of unknown, persistent, but bounded, 


disturbances, 
D= {we lo |||Wllo <1}. 


The performance measure (combined with constraints) is stated as 


lIZllo<y, VweDd. 


o Zz 
——— 
G 
u ee 
—s 
K « 


FIGURE 19.2 General setup. 
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If ® is the linear time-invariant system mapping w to z, then 
IZllo<y, foralweD => ||P <y, 


where 
Ny OO 
lh = max SB 2 Jdi(K)I- 
j= = 


The latter is the expression for the £; norm of the system. In conclusion, the é; norm of a system is the 
peak-to-peak gain of the system and can directly describe time-domain performance specifications. The 
nominal performance problem can be stated as 


inf (sup || ®l]1). 


K stabilizing yw 


19.2.3 Comparison to H.. 


Suppose the exogenous inputs are such that ||w||2 < 1 but are otherwise arbitrary (||w||2 is the energy 
contained in the signal). If the objective is to minimize the energy of the regulated output, then the 
nominal performance problem is defined as 


inf (sup||®w|l2)= inf — sup Omaxl ®(e)]. 
K stabilizing w K stabilizing 9 


Both of these norm minimization problems fall under the same paradigm of minimax optimality. Mini- 
mizing the Ho. norm results in attenuating the energy of the regulated signal but may still result in signals 
that have large amplitudes. Minimizing the £; norm results in attenuating the amplitude of the regulated 
output, and overbounds the maximum energy due to bounded energy inputs because 


Pll. < Vill Olli, 
where m is the number of rows in ®. On the other hand, the following inequality holds: 
Ol]: <2 + Dal Pll.» 


where N is the McMillan degree of the system, and n is the number of columns of ®. The latter bound 
is the tightest possible bound (i.e., equality holds for certain classes of systems) and it shows that the gap 
between these measures can be large if the order of the system is high. 


19.2.4 Prototypes 


The following prototypes have been discussed in [2]. These are representative problems quite common 
in applications. We will use these prototypes to illustrate the significance of the @; design methodology. 


19.2.4.1 Disturbance Rejection 


In the context of foo signals, the disturbance rejection problem is defined as follows: Find a feedback 
controller that minimizes the maximum amplitude of the regulated output over all possible disturbances 
of bounded magnitude. The two-input two-output system shown in Figure 19.3 depicts the particular case 
where the disturbance enters the system at the plant output. Its mathematical representation is given by 


z=Pou+ Wu, 
y = Pou+ Ww. 


The disturbance rejection problem provides a general enough structure to represent a broad class of 
interesting control problems. 
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FIGURE 19.3 A disturbance rejection problem. 


19.2.4.2 Command Following with Saturation 


The command following problem, equivalent to a disturbance rejection problem, is shown in Figure 19.4. 
We will show how to pose this problem in the presence of saturation nonlinearities at the input of the 
plant, as an ¢)-optimal control problem. 


Define the function 
u |u| < Umax 
Sat(u) = 


Umaxsgn(u) |u| > Umax 


Let the plant be described as 
Pu = PoSat(u) 


where Po is LTI. Let the commands be modeled as 
r= Ww where ||wlloo <1. 


The objective is to find a controller K so that y follows r uniformly in time. Keeping in mind the saturation 
function, and in order to stay in the linear region of operation, the allowable control inputs must have 
\|Ulloo < Umax. Let y be the (tracking) performance level desired, and define 


He ~ i 
Z= 
u/Umax 


y = Pou. 


with 


FIGURE 19.4 Command following with input saturation. 
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The problem is equivalent to finding a controller so that 


sup ||Zlloo < 1, 
w 
which is an £;-optimal control problem. 


The above closed loop system will remain stable even if the input saturates, as long as it does so infre- 
quently. The solution to the above problem will determine the limits of performance when the system is 
required to operate in the linear region. Also, the stability for such a system will mean the local stability 
of the nonlinear system. 


19.2.4.3 Saturation and Rate Limits 


In the previous example, actuator limitations may require that the rate of change of the control input be 
bounded. This is captured in the condition 


k)—u(k-1 
OED 2 


where T; is the sampling period. Let 


This condition can be easily incorporated in the objective function by defining z as 


(y—r)/y 
z=} u/Umax 
Weert 


The result is a standard ¢, -optimal control problem. 


19.2.5 Stability and Performance Robustness 


The power of any design methodology is in its ability to accommodate plant uncertainty. The @; norm 
gives a good measure of performance. Because it is a gain over a class of signals, it will also provide a good 
measure for robustness. This is a consequence of the small gain theorem which is stated below [2]. 

Let M be alinear time-invariant system and A be a strictly proper €y.-stable perturbation. The closed-loop 
system shown in Figure 19.5 is €99-stable for all A with || Alle,,—ina < 1 if and only if ||M||, < 1. 

The above result indicates that the ¢; norm is the exact measure of robustness when the perturbations 
are known to be BIBO stable, bounded gain, and possibly nonlinear or time-varying. The result can 
be adapted to derive stability robustness conditions for a variety of plant uncertainty descriptions. We 
describe one such situation below. 


FIGURE 19.5 Stability robustness problem. 
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19.2.5.1 Unstructured Multiplicative Perturbations 


K «—, 


FIGURE 19.6 Multiplicative perturbations. 


Consider the case where the system has input uncertainty in a multiplicative form as in Figure 19.6, 
i.e., let 
Q={P|P=PI+WiAW2) and |lAlle.-ind < 1. 


If a controller is designed to stabilize Pp, under what conditions will it stabilize every system in the set 2? 
By simple manipulations of the closed-loop system, the problem is equivalent to the stability robustness 
of the feedback system in Figure 19.5, with M = W2(I — KP,)~! KP, Wy. In general this manipulation is 
done in a systematic way: Cut the loop at the inputs and outputs of A, and then calculate the map from the 
output of A, w, to the input of A, z. A sufficient condition for robust stability is then given by ||M||1 < 1. 
The resulting two-input two-output description is given by 


y = Pout+ PoWiw, 


Z= W2 Uu. 
This is a standard ¢; minimization problem. 


19.2.5.2 Structured Uncertainty 


In many applications, uncertainty is described in a structured fashion where independent perturbations 
are introduced in various locations of the closed loop system. It turns out that one can derive an exact 
necessary and sufficient condition in terms of a scaled £; norm of the system to guarantee stability 
robustness in the presence of such structured perturbations. It can also be shown that the problem of 
achieving robust performance (where performance is measured in terms of the 2; norm) is equivalent to 
robustly stabilizing a plant with structured uncertainty. In this article, we will not discuss this problem. 
Instead, we refer the reader to [2] for more details. 


19.3 Computability 


Since it is quite hard, in general, to obtain closed form solutions to general optimization problems, we 
need to be precise about the meaning of a “solution.” A closed form solution has two important features: 
the first is the ability to compute the optimal solution through efficient algorithms, and the second is to 
provide a qualitative insight into the properties of the optimal solution. A numerical solution should offer 
both of these ingredients. In particular, it should provide 


1. The exact solution whenever it is possible 
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2. Upper and lower approximations of the objective function when exact solutions cannot be obtained 
and a methodology for synthesizing suboptimal controllers 
3. Qualitative information about the controller, e.g., the order of the controller 


Solutions based on general algorithms even for convex optimization problems offer only approximate 
upper bounds on the solution. To obtain more information, one needs to invoke duality theory. Duality 
theory provides a simple reformulation of the optimization problem from which lower bounds on the 
objective function and, possibly, exact solutions can be found. 


19.3.1 Summary of Results 


To minimize the @; norm, first the Youla parameterization of all stabilizing controllers is invoked. 
The resulting optimization problem can be stated as an infinite-dimensional linear program in the free 
parameter. Two cases occur: 


1. The infinite dimensional LP is exactly equivalent to a finite-dimensional LP. This happens if the 
dimension of the control input is at least as large as the dimension of the regulated variables and 
the dimension of the measured output is at least as large as the exogenous inputs. This means that 
the controller has a lot of degrees of freedom. 

2. If any of the above conditions is violated, then the problem is inherently infinite-dimensional. 
However, duality theory can be used to provide approximate solutions to this problem with 
guaranteed performance levels. 


The details of the computations for both of the above cases can be found in [2]. The most successful 
algorithm for computing solutions for the second case is not based on straightforward approximation, 
but rather on embedding the problem in another that falls under the first case. This procedure generates 
approximate solutions with converging upper and lower bounds, and also provides information about the 
order of the actual optimal controller. Other emerging techniques are based on dynamic programming 
and viability theory and can be found in [1,12]. 


19.4 Flexibility 


Flexibility is the ability to use the design tool to alter a given nominal design so that additional specifica- 
tions are met with minimal expense. Examples of additional specifications include fixed input constraints 
and frequency-domain constraints. The computational cost of alteration should be much less than the 
incorporation of the specification directly in the problem. In addition, it is desirable to maintain the 
qualitative information of the original solution. 

Since the general synthesis problem is equivalent to an infinite dimensional LP, many additional 
specifications (not directly addressed by ¢; norms) can be incorporated as additional linear constraints. 
Frequency-domain constraints can be well-approximated by linear constraints. Below we consider an 
example of adding fixed input constraints to the £; problem. 


19.4.1 ¢, Performance Objective with Fixed Input Constraints 


Consider the case where the specifications given are such that the control signal resulting from a step 
input must be constrained uniformly in time (e.g., to avoid actuator saturation). We want to bound the 
controller response to a step input and at the same time minimize the ¢, norm of the transfer functions 
from the disturbance to both the control signal and plant output. In such a case we augment the basic ¢; 
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FIGURE 19.7 A disturbance rejection problem. 


problem in the following way: 


K(I+PK)"!W 
(I+ PK)-!W 


K stab. 1 


subject to 


< Umax 


|K(I+ PK)" wr loo 


where wy is a unit step input disturbance and Umax is the specified bound. The above modification results 
in adding infinitely many constraints on the sequence K(I + PK)~! (i.e., convolution of a unit step with 
K(I + PK)~'). However, since the peak is typically achieved in early samples, only a finite number of 
constraints must be included (the rest being inactive). This is a particular case of nondecaying template 
constraints which arise frequently in control system design. 


19.4.1.1 Trade-Offs in Design 


We take these specifications a step further by asking the following questions: What are the trade-offs in 
the design? How does the bound on the control signal step response affect the overall performance? And, 
how does it affect the structure of the optimal solution? 

These questions can be readily answered with the ¢; machinery. It amounts to solving a family of mixed 
£, problems parameterized in Umax. Solutions for a range of values of Umax are presented in Figures 19.8 
and 19.9 by showing the performance degradation and the controller order growth as Umax decreases. The 
numerical values are based on a model for the X29 aircraft (for details, see [2]). The following conclusions 
can be drawn from this analysis: 


1. The results present the trade-offs clearly. It is possible to reduce Umax by 50% without losing too 
much performance in terms of the ¢; norm of the system. This implies that the controller can be 
altered to satisfy stricter conditions on the step response without losing the nominal performance. 
The curve in Figure 19.8 also shows the smallest possible achievable Umax. 

2. The trade-offs in the order of the controller are valuable. The trade-off curve in Figure 19.9 shows 
that, by adding two additional states, Umax can be reduced to about 50% of its unconstrained value. 

3. To compute such solutions, the unconstrained problem is solved first. The performance for a step 
input is then checked, and constraints are added only at the time instants where the peak value of 
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FIGURE 19.8 X29: Trade-offs in performance vs. control signal bound, Umax. 


the input u is larger than Umax. This is a simpler problem than incorporating the infinite-horizon 
constraints at all the time instants of the step response. 


Finally, such constraints are hard to deal with by selecting weights and solving an ¢; problem or an Hoo 
problem. The advantage that the ¢; problem has over Hoo is that such constraints can be incorporated in 
the problem, as described earlier, and then solved using the same solution techniques. 


Order of controller 
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FIGURE 19.9 X29: Trade-offs in controller order vs. control signal bound, Umax. 
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19.5 Conclusions 


In this chapter, we gave an overview of the ¢, theory for robust control. The presentation was not detailed. 
However, it was intended to serve as an introduction to a more detailed account of the theory that can 
be found in the book, Control of Uncertain Systems: A Linear Programming Approach, and references 
therein. 

We highlighted three ingredients of the 2; design tool, practicality, computability and flexibility. These 
properties allow for implementing a computer-aided-design environment based on ¢; nominal designs, 
in which the designer has the flexibility to incorporate frequency-domain and fixed-input constraints, 
without losing the qualitative information about the structure of the controllers obtained from the nominal 
designs. Such an environment has proven very powerful in designing controllers for real applications. 
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This chapter gives a brief overview of the structured singular value (1). The j1-based methods discussed are 
useful for analyzing the performance and robustness properties of linear feedback systems. Computational 
software for j1-based analysis and synthesis is available in commercial software products [2,4]. The 
interested reader might also consult the tutorials in references [8,14], and application-oriented papers, 


such as [1,6,13]. 


20.2 Shortcomings of Simple Robustness Analysis 


Many of the theoretical robustness results for single-input, single-output (SISO) systems show that 
if a single-loop system has good robust stability characteristics, and good nominal performance 
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characteristics, then, necessarily, it has reasonably good robust performance characteristics. Unfortu- 
nately, this “fact” is not, in general, true for multiloop systems. Also, for multiloop systems, checking the 
robustness via individual loop-at-a-time calculations can be misleading, because the interactions between 
the deviations are not accounted for in such an analysis. In this chapter, we illustrate these difficulties with 
examples and introduce the structured singular value as an analytical tool for uncertain, multivariable 
systems. 

The first example concerns control of the angular velocity of a satellite spinning about one of its 
principal axes. Its mathematical origins are due to Doyle, and are alluded to in [14]. The closed-loop 
multivariable (MIMO) system is shown in Figure 20.1. 

Set a := 10, and define 


1 s—a@ a(s+1) 1 0 1 1 —-a 
C= aac car ga? |” BS, 0 1’ R= ea a 1° 


A minimal, state-space realization for the plant G is 


0 all 0 
Ac | Be -a 0|0 1 
G= = 20.1 
E | De 1 aloO 0 20:1) 
—a 1/0 0 


In order to assess the robustness margins to perturbations in the input channels into the plant, consider 
the four-input, four-output system, denoted by M, in Figure 20.2. The lines from r; and rz which run 
above K2 and G are included to define the tracking error, e and e2 explicitly. 

Some important transfer functions are 


1 1 


Wy cane Myw,,21 = Myy,z, = s+1 


These imply that the nominal closed-loop system has decoupled command response, with a bandwidth 
of 1 rad/s, the crossover frequency in the first feedback loop is 1 rad/s, with phase margin of 90°, the gain 
margin in the first channel is infinite, the crossover frequency in the second loop is 1 rad/s, with phase 
margin of 90°, and the gain margin in the second channel is infinite. These suggest that the performance 
of the closed-loop system is excellent and that it is quite robust to perturbations in each input channel. 
Yet, consider a 5% variation in each channel at the input to the plant. Referring to Figure 20.3, let 
5; = 0.05, and 82 = —0.05. The output response y(t) to a unit-step reference input in channel 1 is shown 
in Figure 20.4 (along with the nominal responses). Note that the ideal behavior of the nominal system 
has degraded sharply despite the seemingly innocuous perturbations and excellent gain/phase margins in 
the closed-loop system. In fact, for a slightly larger perturbation, 8; = 0.11, 82 = —0.11, the closed-loop 
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FIGURE 20.1 Nominal multiloop feedback system. 
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FIGURE 20.2 Closed-loop system with uncertainty model. 


system is actually unstable. Why do these small perturbations cause such a significant degradation in 
performance? To answer this, calculate the 4 x 4 transfer matrix M represented in Figure 20.2, giving 


1 a s—a? 
s+l1 s+1 s+l 
Wi a 1 s—a@ 21 21 
_ a 
w2)_} sti stl stl 2 = a9 | #2 
e} 1 a S ry ry 
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FIGURE 20.3 Satellite: Closed-loop system with uncertain elements, 8, and 8. 
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Note that the earlier calculations about the closed-loop system yielded information only about the (1, 1), 
(2,2), and (3: 4,3: 4) entries. The notation (3 : 4,3 : 4) denotes the 2 x 2 matrix formed by rows 3 to 4 
and columns 3 to 4 of M. In particular, these entries are all small, in some sense. The neglected entries, 
(1,2), (2,1), (1: 2,3:4), (3:4,1:2) are all quite large, because a= 10. It is these large off-diagonal 
entries, and the manner in which they enter, that causes the extreme sensitivity of the closed-loop 
system’s performance to the perturbations 8; and 8. For instance, with 82 =0, the perturbation 8) 
can only cause instability by making the transfer function (1 — Myizi8i)! unstable. Similarly, with 
5, = 0, the perturbation 5, can only cause instability by making the transfer function (1 — My2,2282)7! 
unstable. Because both Myi,z1 and My2,z2 are “small”, this requires large perturbations, and the single- 
loop gain/phase margins reported earlier are accurate. However, acting together, the perturbations can 
cause instability by making 


1 a sa 


D0) a" 
0 1 a 


s+] 


~s+1{[S 0 
1 0 8 


s+ 


unstable. The denominator of this multivariable transfer function is 


s* + (2481 +82) s+ [1481 +824 (a + 1)8182]. 


By choosing 8; = ~ 0.1, and 8) = —8,, the characteristic equation has a root at s = 0, indicating 


1 
den 
marginal stability. For slightly larger perturbations, a root moves into the right half-plane. The simultane- 
ous nature of the perturbations has resulted in a much smaller destabilizing perturbation than predicted 
by the gain/phase margin calculations. 

In terms of robust stability, the loop-at-a-time gain/phase margins only depended on the scalar transfer 
functions M,,1,z1 and My2,z2, but the robust stability properties of the closed-loop system to simultaneous 
perturbations actually depend on the 2 x 2 transfer function matrix My. Similarly, assessing the robust 
performance characteristics of the closed-loop system involves additional transfer functions ignored in 
the simple-minded analysis. Consider the perturbed closed-loop system in Figure 20.3. In terms of the 


iS Satellite: Nominal and 5% perturbations 
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FIGURE 20.4 Satellite: Nominal (solid) and 5% perturbation responses (dashed). 
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FIGURE 20.5 Perturbed system. 


transfer function matrix M, the perturbed r — e transfer function can be drawn as shown in Figure 20.5. 
Partition the transfer function matrix M into four 2 x 2 blocks, as 


Then the perturbed closed-loop transfer function from r to e can be written as 
e= [Mer +M,A (I —MyA)' Mp| d 


where A is the structured matrix of perturbations, A = diag [8;, 82]. Our initial information about the 
closed-loop system consisted of the diagonal entries of M,, and the entire matrix M2. We have seen that 
the large off-diagonal entries of M); created destabilizing interactions between the two perturbations. In 
the robust performance problem, there are additional relevant transfer functions, Mz; and M2, which 
were not analyzed in the loop-at-a-time robustness tests, or in the r > y nominal command response, 
though it is clear that these transfer functions may play a pivotal role in the robust performance charac- 
teristics of the closed-loop system. 

Hence, by calculating the two single loop-at-a-time robustness tests and a nominal performance test, 10 
of the 16 elements of the relevant 4 x 4 transfer function matrix are ignored. Any test which accounts for 
simultaneous perturbations along with the subsequent degradation of performance must be performed 
on the whole matrix. The point of this example is to show that there are some interesting issues in 
multivariable system analysis and standard SISO ideas that cannot be made into useful analytical tools 
simply by applying loop-at-a-time analysis. The structured singular value (\1), introduced in the next 
section, is a linear algebra tool useful for these types of MIMO system analyses. 


20.3 Complex Structured Singular Value 


This section is devoted to defining the structured singular value, a matrix function denoted by y (-) [5]. 
The notation we use is standard from linear algebra and systems theory. R denotes the set of real numbers, 
C denotes the set of complex numbers, |- | is the absolute value of elements in R or C, R” is the set of 
real n vectors, C” is the set of complex n vectors, ||v|| is the Euclidean norm for v € C”, R”*” is the 
set of n x m real matrices, C’*”™ is the set of n x m complex matrices, I, is the n x n identity matrix, 
and Onxm is an entirely zero matrix. For M € C’*™, M? is the transpose of M, M* is the complex- 
conjugate transpose of M, and G (M) is the maximum singular value of M. For M € C"*", dj (M) is an 
eigenvalue of M, p (M) i= max|)j(M)| is the spectral radius of M, and pr(M) is the real spectral radius 
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of M, pr(M) := max {IX > ER, det (AT-M) = o}, with pr(M) :=0 if M has no real eigenvalues. If 
Me C"*" satisfies M = M*, then M > 0 denotes that M is positive definite, and M 2 means the unique 
positive definite Hermitian square root. For M = M*, then }max(M) denotes the most positive eigenvalue. 

We consider matrices M € C"*". In the definition of \ (M), there is an underlying structure A, (a 
prescribed set of block diagonal matrices) on which everything following depends. This structure may be 
defined differently for each problem depending on the uncertainty and performance objectives. Defining 
the structure involves specifying three things: the total number of blocks, the type of each block, and their 
dimensions. 


20.3.1 Purely Complex |t 


Two types of blocks—repeated scalar and full blocks are considered. Two nonnegative integers, S and 
F, denote the number of repeated scalar blocks and the number of full blocks, respectively. To track the 
block dimensions, we introduce positive integers r1,..., 175; m1,..., mp. The ith repeated scalar block is 
rj X 1, while the jth full block is m; x m;. With those integers given, define A C C"*” as 


A := {diag [81],,,..-,8sIrs. Asti... Aszr]: 8: €C, Asyj ECU, 1 <i<S,1<j<F}. (20.2) 


For consistency among all the dimensions, pan rt Brae mj; must equal n. Often, we will need 
norm bounded subsets of A, and we introduce the notation Ba := {A Ee A:o (A) < 1}. Note that in 
Equation 20.2 all of the repeated scalar blocks appear first, followed by the full blocks. This is done to 
simplify the notation and can easily be relaxed. The full blocks are also assumed to be square, but again, 
this is only to simplify notation. 


Definition 20.1: 


For M € C"*", a (M) is defined 


1 
Pe eer {6 (A): A € A, det (I— MA) =0} aoe) 


unless no A € A makes I— MA singular, in which case wa (M) :=0. 


It is instructive to consider a “feedback” interpretation of 1, (M) at this point. Let M € C”*” be given, 
and consider the loop shown in Figure 20.6. This picture represents the loop equations u = Mv, v= Au. 
As long as I — MA is nonsingular, the only solutions of u, v to the loop equations are u = v = 0. However, 
if I — MA is singular, then there are infinitely many solutions to the equations, and the norms ||u||, || v|| 
of the solutions can be arbitrarily large. Motivated by connections with stability of systems, we call this 


FIGURE 20.6 M -— A interconnection. 
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constant matrix feedback system “unstable”. Likewise, the term “stable” will describe the situation when 
the only solutions are identically zero. In this context then, 1a (M) provides a measure of the smallest 
structured A that causes “instability” of the constant matrix feedback loop in Figure 20.6. The norm of 
this “destabilizing” A is exactly 1/|1, (M). 

Consider M € C>**, 


0.100+0.070i —0.154+ 0.162 0 — 0.560: 0+ 42.000: 
0—0.273i —0.300—0.280i 2.860+0.546i —26.000+ 72.800i 

M := j 0.100 + 0.175 0.077 — 0.1081 —0.400+ 0.210i 5.000 + 3.500: (20.4) 
0+0.002i —0.004—0.002i  0.006+0.011i 0.200 + 0.420: 
0.024 + 0.0281 0+0.027i —0.066+ 0.042: 0 + 0.7001 


to show the dependence of 1, (M) on the set A. Two different block structures compatible (in the sense 
of dimensions) with M are 


Ay = {diag [81, 52, A3] : 81, 89 EC, A3€ Crt and Ad = {diag [81, 82, 83, Ag] ‘ 3; EC, AGE cat - 


The definition yields W,, (M) ~ 8.32, while Uap, (M ) = 2.42. 
An alternative expression for |, (M) follows easily from the definition. 


Lemma 20.1: 


Ha(M) = max (AM) 


Continuity of the function .:C”*”"— R is apparent from this lemma. In general, though, the function 
w:C”*"—> R is not a norm, because it doesn’t satisfy the triangle inequality. However, for any a € C, 
L (aM) = |alp (M ); so it is related to how “big” the matrix is. 

We can relate 1, (M) to familiar linear algebra quantities when A is one of two extreme sets: 


¢ IfA = {81:8 € C} (S=1, F=0, r; =n), then 1a (M) = p (M). 
Proof: The only A’s in A which satisfy the det (I— MA) = 0 constraint are reciprocals of nonzero 
eigenvalues of M. The smallest one of these is associated with the largest (magnitude) eigen- 
value, so, WA (M) =0 (M). 
+ IfA=C"™" (S=0, F=1,m=n), then pa(M) =6 (M). 
Proof: If 6 (A) < amy then 6 (MA) <1, so I— MA is nonsingular. Applying Equation 20.3 
implies 1.4 (M) <6 (M). On the other hand, let u and v be unit vectors satisfying Mv = 
& (M) u, and define A := aq Then o (A) = 1/6(M) and I — MA is obviously singular. 
Hence, A (M) >o (M). 


Obviously, for a general A as in Equation 20.2, {8J, : 8 € C} must be includedin A C C”*". Hence directly 
from the definition of \1, and the two special cases above, we conclude that 
p(M) <ua(M) <o(M). (20.5) 


These bounds by themselves may provide little information about the value of 1, because the gap between 
p and o can be large. To see this, consider the matrix 


w=[p | 


and two different block structures, A; := {8:5 €C} and A) := {A € eo}: 2 with respect to Aj, 
which corresponds to p(M), is 0, independent of a. 1 with respect to Az, which corresponds to o(M), is 
|cx|. 
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The bounds on it can be refined with transformations on M that do not affect a (M ), but do affect p 
and o. To do this, define two subsets, Q, andDa of C”™*”, 


Qa ={QEA: QVQ=]h} (20.6) 


Da = {diag[D1,...,Ds,ds+ilm>.--,4s4+FImp |: Dj € C™™", Dj = DF > 0,ds4j; €R,ds4j >0} (20.7) 


For any A € A,Q€ Qa, andDeEDa, 


QAEQn, QAEA, AQEA G(QA) =G (AQ) =a (A) (20.8) 


D2A = AD? (20.9) 


Theorem 20.1: 
ForallQ€ Qa andDeEDa 
bs (MQ) = Ha (QM) = ita (M) = a (DEMD™?) (20.10) 


Proof 20.1. For all D€ Da and A € A, 


because D commutes with A. Therefore wa (M) = Ua (D)mp-?). Also, for each QE Qa, 
det (I- MA) =0 if and only if det (J —- MQQ*A) =0. Because Q*A = A and 6 (Q*A) =G(A), 
LA (MQ) =A (M ) as desired. The argument for QM is the same. 


Therefore, the bounds in Equation 20.5 can be tightened to 


max p(QM) < max p (AM) 

; , (20.11) 
=v (04) = ings (pi mo?) 

where the equality comes from Lemma 20.1. 

The lower bound, maxgeg (QM), is always an equality [5]. Unfortunately, the quantity p (QM) can 
have multiple local maxima which are not global. Thus local search cannot be guaranteed to obtain 1, but 
can only yield a lower bound. The upper bound can be reformulated as a convex optimization problem, 
so that the global minimum can, in principle, be found. Unfortunately, the upper bound is not always 
equal to 1. For block structures A satisfying 25 + F < 3, the upper bound is always equal to 1, (M), and 
for block structures with 2S + F > 3, matrices exist for which is less than the infimum [5,9]. 
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Convexity properties make the upper bound computationally attractive. The simplest convexity prop- 


erty is given in the following theorem, which shows that the function o (p3mp-?) has convex level 
sets. 


Theorem 20.2: 


Let M € C"*" be given, along with a scaling set Da, andB > 0. Then the set {D €Da:o (Dimp-?) < p| 


is convex. 
Proof 20.2. The following chain of equivalences comprises the proof: 
6 (DMD~?) Sh Be Bike (p-im'!p}mp~!) <p? 
& D-2M*D2D2MD~? —B2I <0 (20.12) 
~ M*DM—§*D <0 


The latter is clearly a convex condition in D because it is linear. 


The final condition in Equation 20.12 is called a Linear Matrix Inequality (LMI) in the variable D. 
In [3], a large number of control synthesis and analysis problems are cast as solutions of LMI’s. 


20.3.2 Mixed |L: Real and Complex Uncertainty 


Until this point, this section has dealt with complex-valued perturbation sets. In specific instances, it 
may be more natural to describe modeling errors with real perturbations, for instance, when the real 
coefficients of a linear differential equation are uncertain. Although these perturbations can be treated 
simply as complex, proceeding with a complex 1 analysis, the results may be conservative. Hence, theory 
and algorithms to test for robustness and performance degradation with mixed (real blocks and complex 
blocks) perturbation have been developed. 

Definition 20.1 of 1 can be used for more general sets A, such as those containing real and complex 
blocks. There are 3 types of blocks, repeated real scalar, repeated complex scalar, and complex full blocks. 
As before in Section 20.3, S and F, denote the number of repeated, complex scalar blocks and the number 
of complex full blocks, respectively. V denotes the number of repeated, real scalar blocks. The block 
dimensions of the real block are denoted by the positive integers f),...,ty. With these integers given, 
and r; and mj as defined in Section 20.3, define A as 


A= {diag [Bilge dpleys 8yigdns- «+s dashes, Avast.» Avto+F] : 
(20.13) 


BL ER35 4; EC Avast ECM, 1 ok <V,1sisS1sj<Fl. 


For consistency among all the dimensions, )> ‘s rut yy rt ae mj; must equal = n. 

The mixed wt function inherits many of the properties of the purely complex wt function, [5,7]. However, 
in some aspects such as continuity, the mixed | problem can be fundamentally different from the complex 
problem. It is now well-known that real 1 problems can be discontinuous in the problem data. Beside 
adding computational difficulties to the problem, the utility of real is doubtful as a robustness measure 
in such cases, the system model is always a mathematical abstraction from the real world, computed to 
finite precision. It has been shown that, for many mixed « problems, jt is continuous. The main idea is 
that, if a mixed \. problem has complex uncertainty blocks that “count,” then the function is continuous. 
From an engineering viewpoint, “count” implies that the complex uncertainty affects the value of ju. This is 
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the usual case because one is usually interested in robust performance problems (which therefore contain 
at least one complex block - see Section 20.5.2), or robust stability problems with some unmodeled 
dynamics, which are naturally covered with complex uncertainty. Purely real problems can be made 
continuous by adding a small amount of complex uncertainty to each real uncertainty (see [1] for an 
example). Consequently, a small amount of phase uncertainty is added to the gain uncertainty. 

The theory for bounding (both lower and upper) mixed real/complex bounds is much more compli- 
cated to describe than the bounding theory for complex jt. The lower bound for the mixed case is a real 
eigenvalue maximization problem. Techniques to solve approximately for a mixed js lower bound using 
power algorithms have been derived, and are similar to those used for a lower bound for complex jt [15]. 
The mixed wt upper bound takes the form of a more complicated version of the same problem, involving an 
additional “G scaling matrix” which scales only the real uncertainty blocks. This minimization, involving 
an LMI expression similar to Equation 20.12, is computed using convex optimization techniques similar 
to those for the purely complex upper bound. See references [7,15] for more computational details. 


20.3.3 Frequency Domain, Robust Stability Tests with |L 


The best-known use of j1 as a robustness analysis tool is in the frequency domain [5,14]. Suppose G(s) is 
a stable, multi-input, multioutput transfer function of a linear system. For clarity, assume G has n, inputs 
and ny outputs. Let A be a block structure, as in Equation 20.2, and assume that the dimensions are such 
that A c C”*"", We want to consider feedback perturbations to G, themselves dynamical systems, with 
the block-diagonal structure of the set A, in Figure 20.7. 

Let S denote the set of real-rational, proper, stable, transfer matrices (of appropriate dimensions, which 
should be clear from context). Associated with any block structure A, let S$, denote the set of all block 
diagonal, stable rational transfer functions, with diagonal block structure as in A. 


Sa -={AeS: A(so) € A for all so € Cy}. 


Theorem 20.3: 


1 
Let B > 0. The loop in Figure 20.7 is well-posed and internally stable for all A € Sq with ||A||og < B’ where 


|A(s)|loo = MaXweR T (AGo)) if, and only if, 
IGlla := sup Ha (Go) <8 
@meR 


In summary, the peak value on the w plot of the frequency response of the known linear system that 
the perturbation “sees” determines the size of perturbations against which the loop is robustly stable. 


-——>| A(s) 


G(s) 


FIGURE 20.7. G— A feedback loop block diagram. 
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Remark 20.1 


If the peak occurs at w = 0, there are systems G where the theorem statement needs to be modified to be 
correct. In particular, it may be impossible to do what the theorem statement implies, that is, construct 
a real-rational perturbation A € S, with || All. = (|IGl| ne and (I — GA)! unstable. Rather, for any 


€ > 0, there will be a real-rational perturbation A € S, with ||All,o = (IG) +e and (I - GA) 
unstable. These facts can be ascertained from results in [9,10]. 

The overall implication of this modification can be viewed in two opposite ways. If the theorem is used 
for actual robustness analysis, the original viewpoint that (\I G|| he is the size of the smallest real-rational 
perturbation causing instability is certainly the “right” mental model to use, because the small correction 
that may be needed is arbitrarily small, and hence of little engineering relevance. On the other hand, if 
Theorem 20.3 is being used to prove another theorem, then one needs to be very careful. 

In the next section, the linear algebra results which extend 1 from a robust stability tool to a robust 
performance tool are covered. These linear algebra results are then applied to give a frequency-domain 
robust performance test in Section 20.5.2. 


20.4 Linear Fractional Transformations and jt 


The use of in control theory depends to a great extent on its intimate relationship with a class of general 
linear feedback loops called Linear Fractional Transformations (LFTs) [11]. This section explores this 
relationship with some simple theorems that can be obtained almost immediately from the definition 
of 1. To introduce these, consider a complex matrix M partitioned as 


My Mi 
M= 20.14 
ie A ( 


and suppose that there is a defined block structure A; compatible in size with Mj; (for any A; € Aj, 
M,,A, is square). For A; € Aj, consider the loop equations 


z=Mywt+Mypd; e=M21w+Mnd; w= Ajz (20.15) 


which correspond to the block diagram in Figure 20.8 (note the similarity to Figure 20.5 in Section 20.2). 
The set of Equations 20.15 is called well posed if, for any vector d, unique vectors w, z, and e exist 
satisfying the loop equations. The set of equations is well-posed if, and only if, the inverse of I — M,, A, 
exists. If not, then depending on d and M, there is either no solution to the loop equations, or there are an 
infinite number of solutions. When the inverse does exist, the vectors e and d must satisfy e = F, (M Al ) d, 
where 
F, (M, Ai) := M2 + My, A\(I— My, A))7 Mp. (20.16) 


F,, (M, A) is called a Linear Fractional Transformation on M by A, and, ina feedback diagram, appears 
as in Figure 20.8. F,, (M, A) denotes that the “upper” loop of M is closed by Aj. An analogous formula 


Ay 


e<—________ d 


FIGURE 20.8 Linear fractional transformation. 
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describes F; (M, Az) which is the resulting matrix obtained by closing the “lower” loop of M with a matrix 
Ad € Ad. 

In this formulation, the matrix M22 is assumed to be something nominal, and A; € Ba, is viewed as a 
norm-bounded perturbation from an allowable perturbation class, A,. The matrices Mj2, M21, and M22 
and the formula F,, (M, A;) reflect prior knowledge showing how the unknown perturbation affects the 
nominal map, M22. This type of uncertainty, called linear fractional, is natural for many control problems 
and encompasses many other special cases considered by researchers. 


20.4.1 Well-Posedness and Performance for Constant LFTs 


Let M be a complex matrix partitioned as 


My M2 
M= 20.17 
iB | ( ) 


and suppose that there are two defined block structures, Aj and Az, compatible in size with My, and M22 
respectively. Define a third structure A as 


dott y | are andre da | (20.18) 
0 A> 


Now there are three structures for which we may compute jt. The notation we use to keep track of this 
is as follows: 1 (-) is with respect to Aj, [2 (-) is with respect to Az, : La (-) is with respect to A. In 
view of this, 1 (Mi1); Wy) (M22) and ta (M) all make sense, though, for instance, |14 (M) does not. For 
notation, let By := {Ai € A, :6 (Ai) < 1 

Clearly, the linear fractional transformation F,, (M, A1) is well-posed for all Ay € Ba, if, and only if, 
1 (M uu) < 1. As the “perturbation” A, deviates from zero, the matrix F,, (M ; Aj) deviates from Mp. 
The range of values for |12 (Fu (M , A)) is intimately related to pa (M), as follows: 


Theorem 20.4: Main Loop Theorem 


The following are equivalent: 


1 (Mi) <1, and 


max 2 (Fu (M, A)) <1 


Wa(M) <1 <=> 


Proof 20.3. The proof of this theorem is based on the definition of js and Schur formulae for determinants 
of block partitioned matrices as in [9]. The Main Loop Theorem forms the basis for all uses of \1 in linear 
system robustness analysis, whether from a state-space, frequency-domain, or Lyapunov approach. 


20.5 Robust Performance Tests Using 1 and Main Loop Theorem 


Often, stability is not the only property of a closed-loop system that must be robust to perturbations. 
Typically there are exogenous disturbances acting on the system (wind gusts, sensor noise) which result 
in tracking and regulation errors. Under perturbation, the effect of these disturbances on error signals 
can greatly increase. In most cases, long before the onset of instability, the closed-loop performance will 
be unacceptably degraded. Hence the need for a “robust performance” test to indicate the worst-case level 
of performance degradation for a given level of perturbations. 
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20.5.1 Characterization of Performance in |t Setting 


Within the structured singular value setting, the most natural (mathematical) way to characterize accept- 
able performance is in terms of MIMO ||-|I,5 (Hoo) norms, discussed in detail in other Chapters (29 and 
40) of this Handbook. In this section, we quickly review the H., norm, and interpretations. 

Suppose T is a MIMO stable linear system, with transfer function matrix T(s). For a given driving 
signal d(t), define é as the output, as in the left-hand diagram of Figure 20.9. 

Assume that the dimensions of T are nz x ng. Let B > 0 be defined as 


B = IIT loo == max [T(jo)]. (20.19) 


A time-domain, sinusoidal, steady-state interpretation of this quantity is as follows: 


Fact: For any frequency @ € R, any vector of amplitudes a € R,,,, and any vector of phases  € R", with 
llall2 < 1, define a time signal 


a, sin (at + 61) 
d(t) = 
Ang sin (at + ng) 


Apply this input to the system T, resulting in a steady-state response és, of the form 


by sin (@t + 1) 
éss(t) = ‘ 
b,, sin (@t + Wn.) 


The vector b € R” will satisfy ||b||, < 8. Moreover, B, as defined in Equation 20.19, is the smallest 
number for which this fact is true for every ||a||, < 1, @, and . 


Multivariable performance objectives are represented by a single, MIMO ||-||,, objective on a closed-loop 
transfer function. Because many objectives are being lumped into one matrix and the associated cost is 
the norm of the matrix, it is important to use frequency-dependent weighting functions, so that different 
requirements can be meaningfully combined into a single cost function. 

In the weighting function selection, diagonal weights are most easily interpreted. Consider the right- 
hand diagram of Figure 20.9. Assume that W; and We are diagonal, stable transfer function matrices, 
with diagonal entries denoted L; and R;. Bounds on the quantity || W TWrlqo will imply bounds about 
the sinusoidal steady-state behavior of the signals d and @(= Td). Specifically, for sinusoidal signal d, 


FIGURE 20.9 Unweighted and weighted MIMO systems. 
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the steady-state relationship between e(= Td), dand || WLTWaIloo follows. The steady-state solution és, 
denoted as 
é sin (@t + 1) 
€ss(t) = : > 


é,, Sin (Ot + Vny) 


satisfies EE 1 |Li(i@)ej|” < 1 for all sinusoidal input signals d of the form, 


d; sin (@t + 44) 
d(t) = 
dy, sin (Gt + Ong) 


satisfying 


if, and only if, ||WiTWrlloo <1. 


20.5.2 Frequency-Domain Robust Performance Tests 


Recall from Section 20.3.1, that if A is a single full complex block, then the function jt, is simply the 
maximum singular value function. We can use this fact, along with the Main Loop Theorem (Theorem 
20.4) and the Hoo notion of performance, to obtain the central robust performance theorem for perturbed 
transfer functions. 

Assume G is a stable linear system, with real-rational, proper transfer function G. The dimension of G 
is nz + ng inputs and ny + ne outputs. Partition G in the obvious manner, so that G1 has n; inputs and 
ny outputs, and so on. Let A C C""*"2 be a block structure, as in Equation 20.2. For A € Sa, consider 
the behavior of the perturbed system in Figure 20.10. 

Define an augmented block structure 


— A 0 - Nd XNne 
ar={{4 n,|: de adrec : 


Ar corresponds to the Az block of the Main Loop Theorem. It is used to compute bounds on 6 (-) of the 
perturbed transfer function F,, (6, A) as A takes on values in Sj. 


-—> A(s) 


FIGURE 20.10 Robust performance LFT. 
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Theorem 20.5: 


Let B > 0. For all A € Sa with ||Alloo < B the perturbed system in Figure 20.10 is well-posed, internally 


stable, and | F, (G, A) I. <B if, and only if 


Glia, = Sup Hay (Gljo)) = B. 


See [14] for details of the proof. The robust performance theorem provides a test to determine if the 


performance of the system F,, (<, A) remains acceptable for all possible norm-bounded perturbations. 


20.5.3 Robust Performance Example 


It is instructive to carry out these steps on a simple example. Here, we analyze the robust stability of 
a simple single-loop feedback regulation system with two uncertainties. The plant is a lightly-damped, 
nominal two-state system with uncertainty in the (2,1) entry of the A matrix (the frequency-squared 
coefficient) and unmodeled dynamics (in the form of multiplicative uncertainty) at the control input. The 
overall block diagram of the uncertain closed-loop system is shown in Figure 20.11. 

The two-state system with uncertainty in the A matrix is represented as an upper linear fractional 
transformation about a two-input, two-output, two-state system H, whose realization is 


0 1 0 0 64 0 0 0 
AS| ag alk oe i cake ae oe Be, ql 


The resulting second order system takes the form 


16 


E(B Si) = 
u (#1) s? + 0.16s + 16(1 + 0.481) 


If we assume that 8; is unknown, but satisfies |3;| < 1, then we interpret the second-order system to have 
40% uncertainty in the denominator entry of the natural frequency-squared coefficient. 

The plant is also assumed to have unmodeled dynamics at the input. This could arise from an unmod- 
eled, or unreliable, actuator, for instance. The uncertainty is assumed to be about 20% at low frequency, 
rising to 100% at 6.5 rad/s. We model it using the multiplicative uncertainty model, using a first-order 
weight, W, = (6.5s + 8)/(s +42). In the block diagram, this is represented with the simple linear frac- 
tional transformation involving 89. 


Uncertain plant 


Zz w. 
j : 5, ; 
Wy dy SD Wy, (|< 
1 as . + 
e< = |< ; I< < i d 
8 Wey fem ee SET Aue fe wap Riate shat Sth Rakeee a AH 
+ 
r ie ce ae 
5 + 


FIGURE 20.11 Robust stability/performance example. 
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The closed-loop performance objective is ||T||., <1, where T is the transfer function from input 
disturbance d and sensor noise n to the output error e, 


ff]-eof] 
n n 


Note that the closed-loop T is a function of both 8; and 82. The scalar blocks which weight the error and 
the noise are used to normalize the two transfer functions that make up T. Finally, for comparison, the 
open-loop system has || Tj ||,9 © 6, and ||T2||,,5 = 0. 

For this example, the controller is chosen as 


—12.56s2 + 17.325 + 67.28 
~~ 53 420.3782 + 136.745 + 179.46" 


Finally, G(s) in Figure 20.12 denotes the closed-loop transfer function matrix from Figure 20.11. The 
dimensions of G are two states, four inputs and three outputs. 


In terms of G, we have 
8, 0 
r=A(o[> 9) 


Hence, using Theorems 20.3 and 20.5, the robust stability and robust performance of the closed-loop 
system can be ascertained by appropriate structured singular value calculations on G (or particular 
subblocks of G). In the next section, we analyze the robust stability and robust performance of the 
closed-loop system for a variety of assumptions on the uncertain elements, 8; and 82. 


20.5.3.1 Analysis 
For notational purposes, partition G(s) into 
Gi Gi2 
G= 20.20 
& | ( 


where G(s) is 2 x 2, and Gy2(s) is 2 x 1. The first two inputs and outputs of G are associated with the 
perturbation structure and the third and fourth inputs and third output correspond to an exogenous mul- 
tivariable disturbance signal, and associated error. For robust stability calculations we are only interested 
in the submatrix G11, and for robust performance calculations the entire matrix G. 
Robust stability calculations are performed with respect to two different block structures: 
Aj := {diag [81,82] : 81,82 € C}, 
Ay = {diag [81,82] : 81 € R82 € C} 
For robust performance calculations, a 2 x 1 full block is appended to A; for the performance calculation, 
yielding Ap C C**?. The two block structures used to evaluate robust performance are: 
Ap, := {diag [81, 82, Ar] : 81,82 € C, Ap € C**'}, 
Ap, := {diag [81, 82, Ar] : 8; €R,82 € C, Ap € C™*'} 


e<m——_ 


FIGURE 20.12 Closed-loop interconnection. 
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All the upper and lower bounds 1 calculations are performed using the jt Analysis and Synthesis 
Toolbox [2]. 


20.5.3.2 Robust Stability 


The robustness of the closed-loop system with respect to linear, complex time-invariant structured per- 
turbations, Aj, is a jt test on G11(jw). The complex, robust stability bounds from the 2 calculation are 
shown in Figure 20.13 (Note that the upper and lower bounds are identical.). The peak jt value is about 
1.29, hence for any A(s) € Sa, stability is preserved as long as || A(s) loo < a and there is a perturbation 
Adest(s), of the correct structure, with || A gest lloo = +h that does cause instability. 

The nominal performance of this system is defined by the Hoo norm of the transfer function G22 is 
|G2allog = 0.22. The maximum singular value of G22 is plotted across frequency in Figure 20.14. The 
robustness and performance measures were originally scaled to be less than 1 when they were achieved. 
Therefore, the system is not robustly stabilized with respect to linear, time-invariant structured complex 
perturbations of size 1, but it achieves the performance objective on the nominal system. 

Recall that the first uncertainty, 5), corresponds to uncertainty in the A(2, 1) coefficient and the second 
uncertainty, 52, corresponds to input multiplicative modeling error. The A(2, 1) coefficient uncertainty 
can be treated as a real uncertainty. This would imply that the magnitude A(2,1) varies between 9.6 
and 22.4. In the initial robust stability analysis, both of these uncertainties were modeled as complex 
perturbations, a potentially more conservative representation of the uncertainty. Let us re-analyze the 
system with respect to Az where 8; is treated as a real perturbation. 

We can analyze the robust stability of the system with respect to one of the uncertainties being real and 
the other uncertainty complex. This is shown in the mixed robust stability plot shown in Figure 20.13. 
Notice that when the A(2, 1) uncertainty is treated as a real perturbation, and the input multiplicative 
uncertainty is complex, the mixed robust stability |. value is reduced from 1.29 to 0.84. Hence the 
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FIGURE 20.13 Complex robust stability (solid) and mixed robust stability (dashed) plots. 
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FIGURE 20.14 Nominal performance plot. 


system is robustly stabilized with respect to real uncertainty in the A(2,1) coefficient and complex input 
multiplicative uncertainty. In this example, it is very conservative to treat the variation in the coefficient, 
A(2, 1), as a complex uncertainty. 


20.5.3.3 Robust Performance 


The closed-loop system under perturbations becomes F,, (G, A). To analyze the degradation of perfor- 
mance due to the uncertainty, we use Theorem 20.5, and the augmented block structure Ap,. The plot 
in Figure 20.15 of Wap, (G(ja)) is shown. The peak is approximately at 1.41. Applying Theorem 20.5 
implies that for any structured A(s) € S,,, with || A(s)|loo < ae the perturbed loop remains stable and, 
the || - ||.o norm of F,, (G, A) is guaranteed to be < 1.41. Also, the converse of the theorem shows that there 


is a perturbation A, whose ||-||,, is arbitrarily close to —_ that causes || F, (4, G) | > 1.41. Therefore 
lo) 


1.41 


robust performance was not achieved. 

Figure 20.15 also shows the results of a mixed w analysis on G(yw) with respect to Ap,. The peak value 
of 1 is 0.99. This implies that for a real perturbation 8; and a finite dimensional, linear time-invariant 
complex perturbation 82(s), stability is preserved and the performance objective achieved. Therefore 
the robust performance objective is achieved when the frequency-squared coefficient is treated as a real 
perturbation and the input multiplicative uncertainty is treated as a complex perturbation. 


20.6 Spinning Satellite: Robust Performance Analysis with | 


Consider the 4 x 4 transfer function M shown (along with its with internal structure) in Figure 20.2. The 
perturbations 8; and 8» enter as shown in Figure 20.3. The appropriate block structure for the robust 
performance assessment of this example is {diag(8 1,82, Ap): 8; € C, Ap € C?%? \. This implies that there 
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FIGURE 20.15 Complex (solid) and mixed (dashed) robust performance plots. 


is independent uncertainty in each of the actuators but that the rest of the model is accurate. Computing 
the structured singular value of 4 (M) across frequency yields a peak of about 11. This implies that a 
diagonal perturbation diag [81,82] of size 1/11 exists so that the perturbed reference-to-error transfer 
function has a singular value peak of approximately 11. This jt analysis clearly detects the poor robust 
performance characteristics of the closed-loop system. In the next section, we turn our attention to design 
techniques which use the structured singular value as a design objective. 


20.7 Control Design via t Synthesis 


Consider the standard linear fractional description of the control problem shown in Figure 20.16. The 
P block represents the open-loop interconnection and contains all of the known elements including the 
nominal plant model, uncertainty structure, and performance and uncertainty weighting functions. The 
Apert block represents the structured set of norm-bounded uncertainty being considered and K represents 
the controller. Apeyt parameterizes all of the assumed model uncertainty in the problem. Three groups of 
inputs enter P, perturbations z, disturbances d, and controls u, and three groups of outputs are generated, 
perturbations w, errors e, and measurements y. The set of systems to be controlled is described by the LFT 


(Fu (ApesP) : Apert € Sip} 


<1, 


The design objective is to find a stabilizing controller K, so that, for all Apert € S Avett’ lA pert lloo 
the closed-loop system is stable and satisfies 


|Fu [Fi (P, K) » Apert | lloo <1. 
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FIGURE 20.16 Linear fractional transformation description of control problem. 


The performance objective involves a robust performance test on the linear fractional transformation 
F, (P, K). To assess the robust performance of the closed-loop system, define an augmented perturbation 
structure, A, 


A 0 
A= {| id | : Apert € Apert» Ar € con 
The goal of 1 synthesis is to minimize overall stabilizing controllers K, the peak value of |L (-) of the 

closed-loop transfer function F)(P, K). More formally, 
min max pa[F)(P, K)(jo)] (20.21) 

K (6) 
stabilizing 

For tractability of the 1 synthesis problem, \1, [-] is replaced by the upper bound for w, 6 [D(.)D~']. The 
scaling matrix D is a member of the appropriate set of scaling matrices D for the perturbation set A. One 

can reformulate this optimization problem as follows: 

} et oe ah 
_ max As o [DuFi(P, K)(go)De | . (20.22) 
stabilizing 

Here, the D minimization is an approximation to the a [F/(P; K)(jw)]. Dw is chosen from the set of 


scalings, D, independently at every w. Hence, 


min min max ¢[DoF/(P,K)(jo)D;']. (20.23) 
K Di. @ 
stabilizing Ded 


The expression max o [-] corresponds to ||[-]|l,9, leaving 
w 


(oe) 


min min |[D.F)(P,K)(-)D']||.- (20.24) 
K dD. 
stabilizing DoeD 
Assume, for simplicity, that the uncertainty block Apert has only full blocks. Then the set Da is of the 


form 
D = {diag [diI,d)I,...,dp_11,I]: dj > 0}. (20.25) 


For any complex matrix M, the elements of Da, which were originally defined as real and positive, can 
take on any nonzero complex values and without changing the value of the upper bound, inf 6 (DMD~ : i 
€ 
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Hence, we can restrict the scaling matrix to be a real-rational, stable, minimum-phase transfer function, 
D(s). The optimization is now 


min min |DF,(P, K)D~ Joo. (20.26) 
K D(s)eD 
stabilizing stablesmin—phase 


This approximation to wt synthesis, is currently “solved” by an iterative approach, referred to as “D — K 
iteration.” 

To solve Equation 20.26, first consider holding D(s) fixed. Given a stable, minimum-phase, real-rational 
D(s), solve the optimizationmin x DF; (P,K \p7} loo. This equation is an Ho. optimization control 

stabilizin 

problem. The solution to the Hoo problem is well-known, consisting of solving algebraic Riccati equations 
in terms of the state-space system. 

Now suppose that a stabilizing controller, K(s), is given, we then solve the following minimization 
corresponding to the upper bound for w. 


sn 1 
min 6 [Do F)(P,K)(go)D;,' | 


This minimization is done over the real, positive D,, from the set Da defined in Equation 20.25. Recall 
that the addition of phase to each d; does not affect the value of o [DuFi(P, K)(jo)D;']. Hence, each 
discrete function, dj, of frequency is fit (in magnitude) by a proper, stable, minimum-phase transfer 
function, dp, (s). These are collected together in a diagonal transfer function matrix D(s), 


D(s) = diag [4e, (s)I, day(s)... dep (SL, 1] 


and absorbed into the original open-loop generalized plant P. Iterating on these two steps comprises the 
current approach to D — K iteration. 

There are several problems with the D — K iteration control design procedure. The first is that we have 
approximated |, (-) by its upper bound. This is not serious because the value of 1. and its upper bound 
are often close. The most serious problem, that the D — K iteration does not always converge to a global, 
or even, local minimum, [14] is a more severe limitation of the design procedure. However, in practice 
the D — K iteration control design technique has been successfully applied to many engineering problems 
such as vibration suppression for flexible structures, flight control, chemical process control problems, 
and acoustic reverberation suppression in enclosures. 


20.8 F-14 Lateral-Directional Control System Design 


Consider the design of a lateral-directional axis controller for the F-14 aircraft during powered approach 
to landing. The linearized F-14 model is found at an angle-of-attack (a) of 10.5 degs and airspeed of 
140 knots. The problem is posed as a robust performance problem with multiplicative plant uncertainty 
at the plant input and minimization of weighted-output transfer functions as the performance criterion. A 
diagram for the closed-loop system, which includes the feedback structure of the plant and controller and 
elements associated with the uncertainty models and performance objectives, is shown in Figure 20.17. 

The overall performance objective is to have the “true” airplane, represented by the dashed box in 
Figure 20.17, respond effectively to the pilot’s lateral stick and rudder pedal inputs. The performance 
objective includes 


1. Decoupled response of the lateral stick, 3).4,, to roll rate, p, and rudder pedals, 8;ygp, to side-slip 
angle, B. The lateral stick and rudder pedals have a maximum deflection of +1 inch. Therefore 
they are represented as unweighted signals in Figure 20.17. 


20-22 
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FIGURE 20.17 F-14 control block diagram. 


2. 


The aircraft handling quality (HQ) response from the lateral stick to roll rate should be a first- 
order system, 5(2)/(s + 2) a The aircraft handling quality response from the rudder pedals to 
side-slip angle should be 2.5558 se 

The stabilizer actuators have +20° and +90°/s deflection and deflection rate limits. The rudder 
actuators have +30° and +125°/s deflection and deflection rate limits. 

The three measurement signals, roll rate, yaw rate and lateral acceleration, are passed through 
second-order antialiasing filters prior to being fed to the controller. The natural frequency and 
damping values for the yaw rate and lateral acceleration filters are 12.5Hz and 0.5, respectively and 
4.1Hz and 0.7 for the roll rate filter. The antialiasing filters have unity gain at DC (see Figure 20.17). 
These signals are also corrupted by noise. 


The performance objectives are accounted for in this framework via minimizing weight transfer func- 
tion norms. Weighting functions serve two purposes in the Hoo and jt framework: they allow the direct 
comparison of different performance objectives with the same norm and they allow incorporating fre- 
quency information into the analysis. The F-14 performance weighting functions include: 


1. 


Limits on the actuator deflection magnitude and rates are included via the Wact weight. Wact is a 
4x 4 constant, diagonal scaling matrix described by Wact = diag(1/90, 1/20, 1/125, 1/30). These 
weights correspond to the stabilizer and rudder deflection rate and deflection limits. 

W, is a3 x 3 diagonal, frequency varying weight used to model the magnitude of the sensor noise. 
Wy = diag(0.025, 0.025, 0.012545) which corresponds to the noise levels in the roll rate, yaw 
rate and lateral acceleration channels. 

The desired 8},,,-to-p and 8;ygp-to-B responses of the aircraft are formulated as a model matching 
problem in the jt framework. The difference between the ideal response of the transfer functions, 
Sistk filtered through the roll rate HQ model and 8,udp filtered through the side-slip angle HQ 
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model, and the aircraft response, p and f, is used to generate an error that is to be minimized. 
The W, transfer function, see Figure 20.17, weights the difference between the idealized roll rate 
response and the actual aircraft response, p. 


__ 0.05s* + 2.90s? + 105.935” + 6.175 + 0.16 
P™ s44.9,19s3 + 30.805? + 18.335 + 3.95 
The magnitude of W, emphasizes the frequency range between 0.06 and 30 rad/s. The desired 


performance frequency range is limited due to a right half-plane zero in the model at 0.002 rad/s, 
therefore, accurate tracking of sinusoids below 0.002 rad/s isn’t required. Between 0.06 and 30 
rad/s, a roll rate tracking error of less than 5% is desired. The performance weight on the B 
tracking error, Wg, is just 2 x Wp. This also corresponds to a 5% tracking error objective. 


All the weighted performance objectives are scaled for an Hoo less than 1 when they are achieved. 
The performance of the closed-loop system is evaluated by calculating the maximum singular value of 
the weighted transfer functions from the disturbance and command inputs to the error outputs, as in 
Figure 20.18. 


20.8.1 Nominal Model and Uncertainty Models 


The pilot has the ability to command the lateral directional response of the aircraft with the lateral stick 
(8istk) and rudder pedals (3,peq). The aircraft has two control inputs, differential stabilizer deflection 
(Sdstab» degs) and rudder deflection (8,yq, degs), three measured outputs, roll rate (p, degs/s), yaw rate 
(r, degs/s) and lateral acceleration (Vac, g’s), and a calculated output side-slip angle (6). Note that B is not 
a measured variable but is used as a performance measure. The lateral directional F-14 model, F-14nom; 
has four states, lateral velocity (v), yaw rate (r), roll rate (p) and roll angle (). These variables are related 
by the state-space equations 
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Sstab os lk Lateral stick command 
Ostab error <———— Closed-loop k_—. rudder pedal command 
Sug error <——_ weighted 


: k————_ p sensor noise 
Srug error <—— performance 


Weighted p error «| _ transfer matrix 


Weighted B error <——__ 


«_— r sensor noise 


}———. y,, sensor noise 


FIGURE 20.18 F-14 weighted performance objectives transfer matrix. 
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The dashed box represents the “true” airplane, corresponding to a set of F-14 plant models define by 
G. Inside the box is the nominal model of the airplane dynamics, F-14,9m, models of the actuators, Gs 
and Gp, and two elements, Wy, and Ag, which parameterize the uncertainty in the model. This type of 
uncertainty is called multiplicative plant input uncertainty. The transfer function W, is assumed known 
and reflects the amount of uncertainty in the model. The transfer function Ag is assumed stable and 
unknown, except for the norm condition, || AG|loo < 1. 

A “first principles” set of uncertainties in the aircraft model would include 


1. Uncertainty in the stabilizers and the rudder actuators. The electrical signals that command deflec- 
tions in these surfaces must be converted to actual mechanical deflections by the electronics and 
hydraulics of the actuators. Unlike the models, this is not done perfectly in the actual system. 

2. Uncertainty in the forces and moments generated on the aircraft, due to specific deflections of 
the stabilizers and rudder. As a first approximation, this arises from the uncertainties in the 
aerodynamic coefficients, which vary with flight conditions, as well as uncertainty in the exact 
geometry of the airplane. 

3. Uncertainty in the linear and angular accelerations produced by the aerodynamically generated 
forces and moments. This arises from the uncertainty in the various inertial parameters of the 
airplane, in addition to neglected dynamics, such as fuel slosh and airframe flexibility. 

4. Other forms of uncertainty that are less well understood. 


In this example, we choose not to model the uncertainty in this detailed manner but rather to lump all 
of these effects together into one, complex full-block, multiplicative uncertainty at the input of the rigid 
body aircraft nominal model. 

The stabilizer and rudder actuators, Gs and Gr, are modeled as first order transfer functions, 25/(s + 
25). Given the actuator and aircraft nominal models (denoted by Gnom(s)), we also specify a stable, 2 x 2 
transfer function matrix W,(s) called the uncertainty weight. These transfer matrices parameterize an 
entire set of plants, G, which must be suitably controlled by the robust controller K. 


G := {Gnom (I + AgWeel) : Ac stable, || Aglloo < 1}. 


All of the uncertainty in modeling the airplane is captured in the normalized, unknown transfer function 
Ag. The unknown transfer function Ag(s) is used to parameterize the potential differences between the 
nominal model Gypom/(s), and the actual behavior of the real airplane, denoted by G. 

In this example, the uncertainty weight Wy is of the form, Wn(s) := diag(w1(s), w2(s))b, for particular 
scalar valued functions w;(s) and w2(s). The w1(s) weight associated with the differential stabilizer input 


is selected to be w1(s) = 204) . The w2(s) weight associated with the differential rudder input is selected 
to be w2(s) = La Hence the set of plants that are represented by this uncertainty weight 


25 0 2(s+4) 
G:= {F-t4non | 38 x | Ih+ Ge 1.5(s+20) | Ag(s) } : Ag(s) stable, || AGlloo = J 


s+25 s+200 


Note that the weighting functions are used to normalize the size of the unknown perturbation Ag. At 
any frequency w, the value of | w) (jo) | and |w. (jo) | can be interpreted as the percentage of uncertainty 
in the model at that frequency. The dependence of the uncertainty weight on frequency indicates that the 
level of uncertainty in the airplane’s behavior depends on frequency. 

The particular uncertainty weights chosen imply that, in the differential stabilizer channel at low 
frequency, there is potentially a 5% modeling error, and at a frequency of 93 rad/s, the uncertainty in 
channel 1 can be as much as 100%, and can get larger at higher frequencies. The rudder channel has more 
uncertainty at low frequency, up to 15% modeling error, and at a frequency of 177 rad/s, the uncertainty 
is at 100%. To illustrate the variety of plants represented by the set G, some step responses of different 
systems from G are shown in Figure 20.19. 
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FIGURE 20.19 Unit-step responses of the nominal model (+) and 15 perturbed models from G. 


The control design objective is a stabilizing controller K so that for all stable perturbations Ac(s), with 
| Aclloo < 1, the perturbed closed-loop system remains stable, and the perturbed weighted performance 


transfer functions has an Hoo norm less than 1 for all such perturbations. These mathematical objectives 


fit exactly into the structured singular value framework. 


20.8.2 Controller Design 


The control design block diagram shown in Figure 20.17 is redrawn as P(s), shown in Figure 20.20. P(s), 
the 25-state, six-input, six-output open-loop transfer matrix, corresponds to the P in the linear fractional 


block diagram in Figure 20.16. 
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FIGURE 20.20 F-14 generalized plant. 
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TABLE 20.1 F-14 D— K iteration information 


Iteration number 1 2 3 4 
Total D-scale order 0 4 4 4 
Controller order 25 29 29 29 
Hoo normachieved 1.562 1.079 1.025 1.017 
Peak 2 value 1.443 1.079 1.025 1.017 


The first step in the D — K iteration control design procedure is to design an Ho (sub)optimal con- 
troller for the open-loop interconnection, P. In terms of the D — K iteration, this amounts to holding the 
d variable fixed (at 1) and minimizing the || - ||». norm of F; (P, K ) over the controller variable K. The 
resulting controller is labeled Kj. 

The second step in the D— K iteration involves solving a analysis problem corresponding to the 
closed-loop system, F; (P, K;). This calculation produces a frequency dependent scaling variable d,,, the 
(1,1) entry in the scaling matrix. In a general problem (with more than two blocks), there would be 
several d variables, and the overall matrix is referred to as “the D-scales.” The varying variables in the 
D-scales are fit (in magnitude) with proper, stable, minimum-phase rational functions and absorbed into 
the generalized plant for additional iterations. These scalings are used to “trick” the Ho. minimization to 
concentrate more on minimizing \ rather than o across frequency. For the first iteration in this example, 
the d scale data is fit with a first-order transfer function. 

The new generalized plant used in the second iteration has 29 states, four more states than the orig- 
inal 25-state generalized plant, P. These extra states are due to the D-scale data being fitted with a 
rational function and absorbed into the generalized plant for the next iteration. Four D— K iterations 
are performed until reaches a value of 1.02. Information about the D— K iterations is shown in 
Table 20.1. All the analysis and synthesis results were obtained with The 1 Analysis and Synthesis Toolbox, 
Version 2.0 [2]. 
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FIGURE 20.21 F-14 robust performance 1 plots with K, and K4 implemented. 
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20.8.2.1 Analysis of the Controllers 


The robust performance properties of the controllers can be analyzed using |. analysis. Robust perfor- 
mance is achieved if, and only if, for every frequency, }1.a (F/(P, K)(jw)) of the closed-loop frequency 
response is less than 1. Plots of \t of the closed-loop system with K, and Ky implemented are shown in 
Figure 20.21. 

The controlled system with K; implemented does not achieve robust performance. This conclusion 
follows from the j1 plot, which peaks to a value of 1.44, at a frequency of 7 rad/s. Because | is 1.44, there 
is a perturbation matrix Ag, so that || Ag|loo = are and the perturbed weighted performance transfer 
functions gets “large.” After four D — K iterations the peak robust performance |1 value is reduced to 1.02 
(Figure 20.21), thereby, nearly achieving all of our robust performance objectives. 

Illustrating the robustness of the closed-loop system in the time domain, time responses of the ideal 
model, the nominal closed-loop system and the “worst-case” closed-loop system from G (using perturba- 
tions of size 1) are shown in Figure 20.22. Controller K4 is implemented in the closed-loop simulations. 


B: ideal (solid), actual (dashed-dot), perturbed (dashed) 
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FIGURE 20.22 Time response plots of the F-14 lateral directional control system. 
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A 1-inch lateral stick command is given at 9 s, held at 1 inch until 12 s, and then returns to zero. The 
rudder is commanded at 1 s with a positive 1 inch pedal deflection and held at 1 inch until 4s. At4sa 
—1-inch pedal deflection is commanded, held to 7 s, and then returned to zero. One can see from the 
time responses that the closed-loop response is nearly identical for the nominal closed-loop system and 
the “worst-case” closed-loop system. The ideal time response for B and p are plot for reference. 


20.9 Conclusion 


This chapter outlined the usefulness of the structured singular value (j1) analysis and synthesis techniques 
in designing and analyzing multiloop feedback control systems. Through examples, we have shown some 
pitfalls with simple-minded analytical techniques, and illustrated the usefulness of the analytic framework 
provided by the structured singular value. We outlined an approach to robust controller synthesis, the 
D — K iteration. As an example, these techniques were applied to the design of a lateral directional control 
system for the F-14 aircraft. 
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21.1 Introduction 


One of the features of modern control theory is the growing presence of algebra. Algebraic formalism 
offers several useful tools for control system design, including the so-called “factorization” approach. 

This approach is based on the input-output properties of linear systems. The central idea is that of 
“factoring” the transfer matrix of a (not necessarily stable) system as the “ratio” of two stable transfer 
matrices. This is a natural step for the linear systems whose transfer matrices are rational, that is, for the 
lumped-parameter systems. Under certain conditions, however, this approach is productive also for the 
distributed-parameter systems. 

The starting point of the factorization approach is to obtain a simple parameterization of all controllers 
that stabilize a given plant. One could then, in principle, choose the best controller for various applications. 
The key point here is that the parameter appears in the closed-loop system transfer matrix in a linear 
manner, thus making it easier to meet additional design specifications. 

The actual design of control systems is an engineering task that cannot be reduced to algebra. Design 
contains many additional aspects that have to be taken into account: sensor placement, computational 
constraints, actuator constraints, redundancy, performance robustness, among many others. There is a 
need for an understanding of the control process, a feeling for what kinds of performance objectives are 
unrealistic, or even dangerous, to ask for. The algebraic approach to be presented, nevertheless, is an 
elegant and useful tool for the mathematical part of the controller design. 


21-1 
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21.2 Systems and Signals 


The fundamentals of the factorization approach will be explained for linear systems with rational transfer 
functions whose input u and output y are scalar quantities. We suppose that u and y live in a space of 
functions mapping a time set into a value set. The time set is a subset of real numbers bounded on the left, 
say R, (the nonnegative reals) in the case of continuous-time systems and Z; (the nonnegative integers) 
for discrete-time systems. The value set is taken to be the set of real numbers R. 

Let the input and output spaces of a continuous-time system be the spaces of locally (Lebesgue) 
integrable functions f from Ry into R, and define a p-norm 


0° 1/p 
fllyy =| [feat] ifi=1 =p <ow, 
0 


Ilfllto = ess sup |f(t)| if p = 00. 
t>0 


The corresponding normed space is denoted by Ly». 

The systems having the desirable property of preserving these functional spaces are called stable. More 
precisely, a system is said to be L» stable if any input u € Lp gives rise to an output y € Lp. The systems 
that are Loo stable are also termed to be bounded-input bounded-output (BIBO) stable. 

The transfer function of a continuous-time system is the Laplace transform of its impulse response g(t), 


(oe) 


G(s) = [se dt. 


0 


It is well known that a system with a rational transfer function G(s) is BIBO stable if and only if G(s) is 
proper and Hurwitz stable, that is, bounded at infinity with all poles having negative real parts. 

In the study of discrete-time systems, we let the input and output spaces be the spaces of infinite 
sequences f = (fo, fi...) mapping Z+ into R and define a p-norm as follows: 


0° 1/p 
fll, = bs i if1 <p <0, 
i=0 


IIflli. = sup lfil ifp = oo. 


i=0 


A discrete-time system is said to be J, stable if it transforms any input u € I, to an output y € Ip. The 
systems that are I,, stable are also known as BIBO stable systems. 
The transfer function of a discrete-time system is defined as the z-transform of its unit pulse response 


(ho, hy, ...), 


oo 
H(z) =) hie", 
i=0 


and it is always proper. A system with a proper rational transfer function H(z) is BIBO stable if and only 
if H(z) is Schur stable, that is, its all poles have modulus less than one. 

Of particular interest are discrete-time systems that are finite-input finite-output (FIFO) stable. Such 
a system transforms finite-input sequences into finite-output sequences, its unit pulse response is finite, 


and its transfer function H(z) has no poles outside the origin z = 0, that is, H(z) is a polynomial in z~!. 
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21.3 Fractional Descriptions 


Consider a rational function G(s). By definition, it can be expressed as the ratio 


B(s) 
G(s) = A) 
of two qualified rational functions A and B. 
A well-known example is the polynomial description, in which case A and B are coprime polynomials, 
that is, polynomials having no roots in common. 
Another example is to take for A and B two coprime, proper and Hurwitz-stable rational functions. 
When G(s) is, say, 


stl 
G(s) = ; 
6) +l 
then one can take , 
s+] s+1 
A ==... B ——— 
(s) (En? (s) G41) 


where i > 0 is areal number. We recall that two proper and Hurwitz-stable rational functions are coprime 
if they have no infinite nor unstable zeros in common. Therefore, in the example above, the denominator 
of A and B can be any strictly Hurwitz polynomial of degree exactly 2; if its degree is lower, then A would 
not be proper and if it is higher, then A and B would have a common zero at infinity. The set of proper 
and Hurwitz-stable rational functions is denoted by Ry(s). 

The proper rational functions H(z) arising in discrete-time systems can be treated in a similar manner. 
One can write 

Be) 
~ A(z) 
where A and B are coprime, Schur-stable (hence proper) rational functions. Coprimeness means having 
no unstable zeros (i.e., in the closed disc |z| >1) in common. For example, if 


H(z) 


1 
H(z) = —., 
(z) FA 
then one can take 
A@)=2—, Bw) 
Zz) = —, — 
zZ—h zZ—h 


for any real number X such that |X| < 1. The set of Schur-stable rational functions is denoted by Rs(z). 
The particular choice of \=0 in the example above leads to 


z—1 1 
=1l—z!, Bz)=-=21. 
Zz 


A(z) = 


In this case, A and B are in fact polynomials in z~!. 


21.4 Feedback Systems 


To control a system means to alter its dynamics so that a desired behavior is obtained. This can be done 
by feedback. A typical feedback system consists of two subsystems, S; and Sz, connected, as shown in 
Figure 21.1. 

In most applications, it is desirable that the feedback system be BIBO stable in the sense that whenever 
the exegenous inputs u; and uz are bounded in magnitude, so too are the output signals y; and yp. 
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FIGURE 21.1 Feedback system. 


In order to study this property, we express the transfer functions of S; and S2 as ratios of proper stable 
rational functions and seek for conditions under which the transfer function of the feedback system is 
proper and stable. 

To fix ideas, consider continuous-time systems and write 


B(s) Y(s) 
~Atsy 2 Xs)? 


S} 


where A, B and X, Y are two couples of coprime rational functions from Ry(s). The transfer matrix of 
the feedback system 
Sj S1S2 
yy 1—S$)S.. 1—S,S2 uy 
al ~ | SS, S2 | 


1—S$)S.. 1—S,S2 


pal _ 1 BX —BY uy 
2} AX+BY|—-BY —AY||m]’ 
We observe that the numerator matrix has all its elements in Ry(s) and that no infinite or unstable 
zeros of the denominator can be absorbed in all these elements. We therefore conclude that the transfer 


functions belong to Ry(s) if and only if the inverse of AX+ BY is in Ry(s). 
We illustrate with the example where S, is a differentiator and S» is an invertor such that 


is then given by 


Sy(s)=s, S2(s)=—1. 


We take 
1 Ss 
A(s) = , Bis)= 
(s) str (s) sth 
for any real } > 0 and 
X(s)=1, Y(s)=1. 
Then 
sth 
AX +BY) \(s)= 
(AX +BY) “(s) a 


resides in Rz(s) and hence the feedback system is BIBO stable. 

The above analysis applies also to discrete-time systems; the set Ry(s) is just replaced by Rs(z). However, 
we note that any closed loop around a discrete-time system involves some information delay, no matter 
how small. Indeed, a control action applied to S; cannot affect the measurement from which it was 
calculated in Sy. Therefore, either S(z) or S2(z) must be strictly proper; we shall assume that it is Sj (z) 
that has this property. 
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To illustrate the analysis of discrete-time systems, consider a summator S; and an amplifier Sp, 


1 
Si(z) = a So(z) = —k. 
Taking 
A@=2—, Bey=— 
a = « a 


for any real } in magnitude less than 1 and 
X(z)=1, Y(z)=k, 


one obtains 
Z—k 


z—-(1-k) 


Therefore, the closed-loop system is BIBO stable if and only if |1 — k| < 1. 

To summarize, the fractional representation used should be matched with the goal of the analysis. The 
denominators A, X and the numerators B, Y should be taken from the set of stable transfer functions, 
either Ry(s) or Rs(z), depending on the type of the stability studied. This choice makes the analysis more 
transparent and leads to a simple algebraic condition: the inverse of AX + BY is stable. Any other type of 
stability can be handled in the same way, provided one can identify the set of the transfer functions that 
these stable systems will have. 


(AX + BY) !(z) = 


21.5 Parameterization of Stabilizing Controllers 


The design of feedback control systems consists of the following: given one subsystem, say S;, we seek 
to determine the other subsystem, S2, so that the resulting feedback system shown in Figure 21.1 meets 
the design specifications. We call S; the plant and S2 the controller. Our focus is first on achieving BIBO 
stability. Any controller Sz that BIBO stabilizes the plant S; is called a stabilizing controller for this plant. 
Suppose S; is a continuous-time plant that gives rise to the transfer function 
B(s) 

Si(s) = AG) 
for some coprime elements A and B of Ry(s). It follows from the foregoing analysis that a stabilizing 
controller exists and that all controllers that stabilize the given plant are generated by all solution pairs X, 
Y with X 4 0 of the Bézout equation 

AX+BY=1 


over Ry(s). There is no loss of generality in setting AX + BY to the identity rather than to any rational 
function whose inverse is in Ry(s): this inverse is absorbed by X and Y and therefore cancels in forming 


_ Y(s) 
Sy(s) = ~) 


The solution set of the equation AX + BY = 1 with A and B coprimes in Ry(s) can be parameterized as 
X=X'+BW, Y=Y'-AW, 


where X’, Y’ represent a particular solution of the equation, and W is a free parameter, which is an 
arbitrary function in Ry(s). 
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The parameterization of the family of all stabilizing controllers S2 for the plant S; now falls out almost 
routinely: 
Y’(s) — A(s) W(s) 
X'(s) + B(s) W(s)’ 
where the parameter W varies over Ry(s) while satisfying X’+ BW # 0. 


Sa(s) 


In order to determine the set of all controllers Sj that stabilize the plant S;, one needs to do two things: 
(1) express S;(s) as a ratio of two coprime elements from Ry(s) and (2) find a particular solution in Ry(s) 
of a Bézout equation, which is equivalent to finding one stabilizing controller for S}. Once these two steps 
are completed, the formula above provides a parameterization of the set of all stabilizing controllers for 
S,. The condition X’ + BW # 0 is not very restrictive, as X’ + BW can identically vanish for at most one 
choice of W. 

As an example, we shall stabilize an integrator plant S). Its transfer function can be expressed as 


where s+ 1 is an arbitrarily chosen Hurwitz polynomial of degree one. Suppose that using some design 
procedure we have found a stabilizing controller for S,, namely 


So(s) = —1. 
This corresponds to a particular solution X’ = 1, Y’ = 1 of the Bézout equation 


Ss 1 
X+—Y=1. 
s+] s+l 


The solution set in Ry(s) of this equation is 
X()=1+——W6), ¥(s)=1- WO) 
= —— : s)=1-—-—W(s). 
= stl : stl 


Hence, all controllers Sz that BIBO stabilize S; have the transfer function 


1—(s/s+1)W(s) 
14+ (1/s+1)W(s)’ 


S2(s) = 


where W is any function in Ry(s). 
It is clear that the result is independent of the particular fraction taken to represent S;. Indeed, ifs +1 
is replaced by another Hurwitz polynomial s + in the above example, one obtains 


n= (s/s +r) W"(s) 
14+. (1/s+)W"(s)’ 


So(s) = 


which is the same set when 


ne 2 sth 
w= (=) WO 


21.6 Parameterization of Closed-Loop Transfer Functions 


The utility of the fractional approach derives not merely from the fact that it provides a parameterization 
of all controllers that stabilize a given plant in terms of a free parameter W, but also from the simple 
manner in which this parameter enters the resulting (stable) closed-loop transfer matrix. 
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In fact, 


yi} | BXX’+BW) —B(Y’— AW) uy 
y2| | -B(Y’—- AW) A(Y'—AW) u2 |’ 
and we observe that all four transfer functions are affine in the free parameter W. 
This result serves to parameterize the performance specifications, and it is the starting point for the 
selection of the best controller for the application at hand. The search for S2 is thus replaced by a search 


for W. The crucial point is that the resulting selection/optimization problem is linear in W, whereas it is 
nonlinear in S). 


21.7 Optimal Performance 


The performance specifications often involve a norm minimization. 

Let us consider the problem of disturbance attenuation. We are given, say, a continuous-time plant Sj 
having two inputs: the control input u and an unmeasurable disturbance d (see Figure 21.2). The objective 
is to determine a BIBO stabilizing controller Sz for the plant S; such that the effect of d on the plant output 
y is minimized in some sense. 

We describe the plant by two transfer functions 

Siu(s) = nat Sja(s) = not 
where A, B, and C is a triple of coprime functions from Ry(s). The set of stabilizing controllers for S; is 
given by the transfer function 
Y'(s) — A’(s) W(s) 
X"(s) + BY(s)W(s)’ 


where A’, B’ represent a coprime fraction over Ry(s) for Siu, 


So(s) = 


B(s) _ B’(s) 
A(s)_ A’(s) 


and X’, Y’ represent a particular solution over Ry(s) of the equation 
A'X+B'Y =1, 


such that X’+ B’W £0. 
The transfer function, G(s), between d and y in a stable feedback system is 


Sid 


G = — =C(X'+B'Ww 
1 — SiyS2 eo 


and it is affine in the proper and Hurwitz-stable rational parameter W. 


FIGURE 21.2 Disturbance attenuation. 
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Now suppose that the disturbance d is any function from Loo, that is, any essentially bounded real 
function on R;4. Then 


Iylltoo WGI M4 llt.0» 


where 


Gln = f igwiae 
0 


and g(t) is the impulse response corresponding to G(s). The parameter W can be used to minimize the 
norm ||G||; and hence the maximum output amplitude. 
If d is a stationary white noise, the steady-state output variance equals 


Ey? = ||G\|5Ed’, 
where 


zh topes ac 
iai= [ oP a= $ a-sae) ads 
0 


The last integral is a contour integral up the imaginary axis and then around an infinite semicircle 
in the left half-plane. Again, W can be selected so as to minimize the norm ||G||2, thus minimizing the 
steady-state output variance. 

Finally, suppose that d is any function from Ly, that is, any finite-energy real function on R;. Then one 
obtains 


IIyllz> < IIGllooll4llz,» 


where 


IlGlloo = sup |G(s)]. 
Res>0 


Therefore, choosing W to make the norm ||G||,5 minimal, one minimizes the maximum output energy. 
The above system norms provide several examples showing how the effect of the disturbance on the 
plant output can be measured. The optimal attenuation is achieved by minimizing these norms. 
Minimizing the 1-norm involves a linear program while minimizing the oo-norm requires a search. 
The 2-norm minimization has a closed-form solution, which will be now described. 
We recall that 


G(s) = P(s) + Q(s) W(s), 


where P = CX’ and Q= CB’. The norm ||G||z is finite if and only if G is strictly proper and has no poles 
on the imaginary axis; hence we assume that Q has no zeros on the imaginary axis. We factorize 


Q= Qap Qmp> 


where Qap satisfies Qap(—-s)Qup(s) = 1 (the so-called all-pass function) and Qmp is such that Gp is in 
Ry(s) (the so-called minimum-phase function); this factorization is unique up to the sign. Let Q/,, denote 
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the function Qip(s) = Qap(-s). Then 
GI = |P + QWII5 
= ||QipP + Qnp WII5. 


Decompose Q/7,,P as 
OP = (QapP)st + (QipP)un 


where (QipP)st is in Ry(s) and (QipP)un is unstable but strictly proper; this decomposition is unique. 
Then the cross-terms contribute nothing to the norm and 


[GHZ = I(QipP)unllz + II(QipP)st + Qmp WII3- 
Since the first term is independent of W, 


min ||Gll2 = I(QipP)unll2 


and this minimum is attained by 


QP a 
Oa 
Here is an illustrative example. The plant is given by 
s—2 
Stu(s) = oe Sig(s) =1 


and we seek to find a stabilizing controller Sy such that 


S 
G(s) = 1d(s) 
1 — S},,(s)S2(s) 
has minimum 2-norm. 
We write 
AGN, BS. eS i 
sj=1, $s) = — , = 
stl 


and find all stabilizing controllers first. Since the plant is already stable, these are given by 


—W(s) 
14+ (s—2)/(s+1)W(s)’ 


S2(s) = 


where W isa free parameter in Ry(s). 


Then 
s—2 
G(s) = 1+ ——W(s), 
(s)=1+ ae (s) 
so that 
s—2 
P(s) = 1, ne 
(s) Q(s) a 
Clearly, 
s—2 s+2 
Qap(s) = Gay Qinp(s) = ed 
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and 
s+2 4 
* (s)P(s) = =a : 
ap (S)P(s) 5 es 
Therefore, 
4 s+2 
G\? = 244 Wir, 
Gls Gla I a IIs 
so that the least norm 
4 4 
min ||G||2 = = = 2 
ni Glo Fle gle 
is attained by 
s+1 
W(s) = — : 
(s) Ss 


21.8 Robust Stabilization 


The actual plant can differ from its nominal model. We suppose that a nominal plant description is 
available together with a description of the plant uncertainty. The objective is to design a controller that 
stabilizes all plants lying within the specified domain of uncertainty. Such a controller is said to robustly 
stabilize the family of plants. 

The plant uncertainty can be modeled conveniently in terms of its fractional description. To fix ideas, 
we shall consider discrete-time plants factorized over Rs(z) and endow Rs(z) with the oo-norm: for any 
function H(z) from Rs(z), 


|Z loo = sup |H(z) 
|z|>1 


For any two such functions, H(z) and H2(z), we define 


A 
IIL Hallloo = | | is loo 


= sup (|Ai(z)|? + |Ho(z)|?)/?. 
|z|>1 


Let Sjo be a nominal plant giving rise to a strictly proper transfer function 


where A and B are coprime functions from Rs(z). We denote S;(A, B, 1) the family of plants having 
strictly proper transfer functions 
7 B(z) + AB(z) 


S= Fey aRAG)’ 


where AA and AB are functions from Rs(z) such that 
[AA AB]]loo <b 


for some nonnegative real number |. 
Now, let S be a BIBO stabilizing controller for S19. Therefore, 


Y’—-AW 


eer 
X’+ BW 
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where AX’ + BY’ = 1 and W is an element of Rs(z). Then S2 will BIBO stabilize all plants from S,(A, B, 
\) if and only if the inverse of 

7 

(A+ AA)(X’ + BW) + (B+ AB)(Y’—AW)=1+[AA_ AB] E yd 


is in Rs(z). This is the case whenever 


X'+ BW 
[AA AB] | lloo < 15 


Y’—-AW 


thus, we have the following condition of robust stability: 


X'+BW rat 
BI Tyr aw] los” 


The best controller that robustly stabilizes the plant corresponds to the parameter W that minimizes 
the co-norm above. This requires a search; closed-form solutions exist only in special cases. One such 
case is presented next. 

Suppose the nominal model 


1 
Sio(z) = 1 
has resulted from : 
Z+ 
S = —____ 
1) (z—1)(z—8) 


by neglecting the second-order dynamics, where § > 0 and 0 < « < 1. Rearranging, 


(1/z)+ (/z) (8+ €/z— 8) 


Sek (z—1/2) 


and one identifies 


18 
N= ihe “pees 
y ey mies oF 


Hence, 


s+e 
[AA AB]lloo = ne 


and the true plant belongs to the family 


z—l1 1 s+e 
si( ; ; ). 
Zz Zz l-« 


All controllers that BIBO stabilize the nominal plant Si are given by 


1—(z—1/z)W(z) 
1+ (1/z)W(z) 


S2(z) = 


where W is a free parameter in Rs(z). Which controller yields the best stability margin against 8 and ¢? 
The one that minimizes the oo-norm in 


aang 
l-« z—l1 
1 
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Suppose we wish to obtain a controller of McMillan degree zero, S2(z) = —K. Then 


W(z)=(1 GK 


and |1 — K| < 1. The norm 


1 
1+-W ; 
= = ¥ (1+ K*)| — lle 
z—1 z—(1—K) 
1- Ww 
z [e-e) 


attains the least value of ,/2 by K = 1, which corresponds to W(z) = 0. It follows that the controller 


S2(z) =—-1 
stabilizes all plants S,(z) for which 

s+e 2 1 

l-e 2 


21.9 Robust Performance 


The performance specifications often result in divisibility conditions. A typical example is the problem 
of reference tracking. 
Suppose we are given a discrete-time plant S,, with transfer function 
B(z) 


Si(z) = A® 


in coprime fractional form over Rs(z), together with a reference r whose z-transform is of the form 


E(z) 
f=; 
D(z) 
where only D is specified. We recall that S,(z) is strictly proper. The objective is to design a BIBO 
stabilizing controller Sz such that the plant output y asymptotically tracks the reference r (see Figure 21.3). 


The controller can operate on both r (feedforward) and y (feedback), so it is described by two transfer 
functions 


where X, Y, and Z are from Rs(z). 


FIGURE 21.3 Reference tracking. 
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The requirement of tracking imposes that the tracking error 


BZ E 
e=r—y=(1 
AX+BY/ D 


belong to Rs(z). Since AX + BY has inverse in Rs(z) for every stabilizing controller and E is unspecified, 
D must divide 1 — BZ in Rs(z). Hence, there must exist a function V in Rs(z) such that 1 —- BZ = DV. 
Therefore, S2 exists if and only if B and D are coprime in Rs(z), and the two controller transfer functions 
evolve from solving the two Bézout equations 


AX + BY =1, 
DV+BZ=1, 


where the function V serves to express the tracking error as 
e= VE. 


The reference tracking is said to be robust if the specifications are met even as the plant is slightly 
perturbed. We call S19 the nominal plant and Sj (A, B, |) the neighborhood of S19 defined by 


B+ AB 


S,= : 
A+AA 


where AA and AB are functions of Rs(z), such that 
[AA AB]]loo <b 


for some nonnegative real number |. We recall that all S| (z) are strictly proper. 
Now A+ AA and B+ ABare not specified, but (A + AA)X + (B+ AB)Y still has inverse in Rs(z); call 


it U. We have i RO Por ‘ 
e= zs X+ a (Y-Z)) x. 
U U D 
Hence, for robust tracking, D must divide both X and Y — Z in Rs(z). But, it is sufficient that D divides 
X; this condition already implies the other one as can be seen on subtracting the two Bézout equations 
above. 
We illustrate on a discrete-time plant S; given by 


1 
Si(z) = 7_2 


whose output is to track every sinusoidal sequence of the form 


az+b 
2—z+l 


where a, b are unspecified real numbers. Taking 


Z=2 


2 
A(z)= —.,  B(z)= 7 D(z) = cae a 
Zz Zz 


g2 


and solving the pair of Bézout equations 


z—2 1 
X(z)+ -Y(z)=1, 
z z 


z—-z+1 


5 Viz) + 172) = 1 
Zz Zz 
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yields the tracking controllers in parametric form 


2 —(z-—2/z)Wi(z) 
1+ (1/z)Wi(z) 
(z—1/z)— (22—zt 1/z*)Wo(z) 
14+ (1/z)Wi(z) 


Say(z) = 


Sor(z) = 


for any elements W1, W2 of Rs(z). The resulting error is 


az+b 
ze 


e= E + Wac)| 
Zz 


Not all of these controllers, however, achieve a robust tracking of the reference. The divisibility condi- 
tion is fulfilled if and only if Wj is restricted to 


z-1l 2-—z+l 
Wi(z)= . t oe 


W(z), 


where W is free in Rg(z). 
It is to be noted that the requirement of asymptotic tracking leaves enough degrees of freedom to meet 
additional design specifications. 


21.10 Finite Impulse Response 


Transients in discrete-time systems can settle in finite time. Systems having the property that any input 
sequence with a finite number of nonzero elements produces an output sequence with a finite number 
of nonzero elements have been called FIFO stable. We recall that a system with proper rational transfer 
function H(z) is FIFO stable if and only if H(z) is a polynomial in z~!. 
Let us consider the feedback system shown in Figure 21.1 and focus on achieving FIFO stability. To 
this end, we write the transfer function of the plant as 
B(z) 


$1(z) = wey 


where this time A and B are coprime polynomials in z~!. We recall that the plant incorporates the 

necessary delay so that S;(z) is strictly proper. Repeating the arguments used to design a BIBO stable 

system, we conclude that all controllers S; that FIFO stabilize the plant S; have the transfer function 
Y(z) 


S2(z) = Xa’ 


where X, Y represent the solution class of the polynomial Bézout equation 
AX+BY=1., 


In particular, if X’ and Y’ define any FIFO stabilizing controller for S,, the set of all such controllers can 
be parameterized as 
So(z) = Y’—AW 
Be BW 


where W(z) is a free polynomial in z~!. 


It is a noteworthy fact that the parametric expressions for the sets of BIBO stable and FIFO stable 
controllers are the same; the only difference is that the free parameter of FIFO stabilizing controllers is 
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permitted to range over only the smaller set of polynomials in z~!, whereas in BIBO stabilizing controllers 
it is permitted to range over the larger set of Schur-stable rational functions in z. Indeed, FIFO stability is 
more restrictive than BIBO stability. 

The design options offered by FIFO stability are remarkable. The parameter W can be selected so as 
to minimize the McMillan degree of S2, or to achieve the shortest impulse response of the closed-loop 
system. Various norm minimizations can also be performed. 

A well-known example is the deadbeat controller. We consider a double-summator plant with transfer 
function 


1 
Si(z) = @=1 


and interpret the exogenous inputs u; and uz as accounting for the effect of the initial conditions of S; 
and Sy. The requirement of FIFO stability is then equivalent to achieving finite responses y; and yp for all 
initial conditions. Since in this case 


A(z)=(1—z7')*, Biz) =z? 


and the Bézout equation 
(Q—z!X(2)+z27Y(z)=1 


has a particular solution 
X(zy=lte2, Y"(z)=3—2271, 


we obtain all deadbeat (or FIFO stabilizing) controllers as 


3-227! -(1-—27!) W(z) 
14+2z-!+2-2W(z) 


S2(z) = 


The deadbeat controller ofleast McMillan degree (=1) is obtained for W(z) = 0. The choice W(z) = —3 
leads to a deadbeat controller that rejects step disturbances uj (hence, persistent) at the plant output y; in 
finite time. And when 1 is a stationary white noise, then W(z) = 2.5 minimizes the steady-state variance 
of y; among all deadbeat controllers of McMillan degree 2. 


21.11 Multivariable Systems 


Up until now we have considered only single-input single-output (SISO) plants and controllers. In the 
case of multiple inputs and/or outputs, the input-output properties of linear systems are represented by 
a matrix of transfer functions. The additional intricacies introduced by these systems stem mainly from 
the fact that the matrix multiplication is not commutative. 

Consider a rational transfer matrix G(s) whose dimensions are, say, m x n. Then it is always possible 
to factorize G as follows: 


G(s) = Br(s)AR (s) 
= Ay '(s)BL(s); 


where the factors Br, Ar and Az, By are, respectively, m x n,n x nandm x m,m x n matrices of qualified 
rational functions, say from Ry(s), such that 


Ar, Bare right coprime, 


Az, By are left coprime. 


These “matrix fractions” are unique except for the possibility of multiplying the “numerator” and the 
“denominator” matrices by a matrix whose determinant has inverse in Ry(s). That is, if G(s) can also be 
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expressed as 
G(s) = Ba(s)AR *(s) 
= A, \(s)By(s), 
where the factors are matrices of functions from Ry(s), such that 
Ak, Bp are right coprime, 
Aj, By are left coprime, 
then 
A’R(s) = Ar(s)UR(s),  BYa(s) = Br(s)UR(s), 
A’1(s) = UL(s)AL(s),  B'i(s) = Ux(s)Bi(s) 


for some matrices Ug and Uy over Rxy(s), whose determinants have stable inverses in Ry(s). 
Analogous results hold for discrete-time systems. To illustrate, consider the transfer matrix 


1 2=z 


= Qi. 
G(z) = Zk) (24% 


z—1 z—1 


and determine its left and right coprime factorizations over Rs(z). One obtains, for instance, 


1 2.=% -1 
- = 1 0 
G@=|% 277% ieee 
0 = = 
25. Z—h 
-1 
al aR 
Zz 
= z—1 
0 1 1 
zZ—W Z—-wW Zw 


for any real } and jt with modulus less than one. 
Let us now consider the feedback system shown in Figure 21.1 where S; and S2 are multivariable 
systems and analyze its BIBO stability. We therefore factorize the two transfer matrices over Ry(s); 


Si(s) = Br(s)Ap'(s) = Az '(s)Bx(s); 
So(s) = —X7,*(s)¥i(s) = —Yr(s)Xp (5), 


where the two pairs Ar, Br and Xp, Yp are right coprimes while the two pairs Ay, By and Xz, Yz are left 
coprimes. The transfer matrix of the feedback system 


yl _ [| S\@—S28))7! $I $28))7!S uy 
y2|~ [Sx —$1S2)*S; So — $$)? u2 


yi} [ Br(X,Ar+Y,Br)"'X,  —Br(XrAr+ YiBr)7!Y1 uy 
yo| |I—Ar(X,Ar+ YiBr)'X_, —Ar(XtAr + YiBr) Yi uz 


then reads 


or alternatively 


yi] | Xp(ALXe+ByYr)'B, Xp(ArXpr+BiYp)'Ap —1 uy 
ya} |-Yr(ALXr+BiYr)'Bp —Yr(AtXr+BiYr)'At u2 |° 


The feedback system is BIBO stable if and only if this transfer matrix has entries in Ry(s). We therefore 
conclude that the feedback system is BIBO stable if and only if the common denominator X,Ar + YrBr, 
or alternatively A; Xp + By, Yr, has inverse with entries in Ry(s). 
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A parameterization of all controllers S, that BIBO stabilize the plant S; is now at hand. Given left and 
right coprime factorizations over Ry(s) of the plant transfer matrix 


S} = BrAp' = A; BL, 
we select matrices X;, Y; and Xp, Yp with entries in Ry(s), such that 
XlAg+Y/Br=I, A,X’p+B,Y,=1. 
Then the family of all stabilizing controllers has the transfer matrix 


So = —(Xi, + WB)‘ (Yi, — WLAL) 
= —(Yp—ArWr)(Xp+ BrWr)', 


where W, is a matrix parameter whose entries vary over Ry(s) such that X; + W,By, is nonsingular, and 
Wr is a matrix parameter whose entries vary over R7;(s) such that Xp + Br Wp is nonsingular. 
Asan example, determine all BIBO stabilizing controllers for the discrete-time plant considered earlier, 
with the transfer matrix 
1 2=Z 


= 2 
CAE ou Zz ; Zz 


z—l z—I 


The left and right coprime factors over Rs(z) can be taken as 


1 0 oe = 
Ar(z) = go le Bea: 
= 0 1 
z Zz 
and 
i 2a Ge ee 
AL(z) = 0 z-1|, BLo= 1 ra 
z ZZ 


The Bézout equations 
X,AR + Y;Br =I, ALXp +BLYp =I 


xio=[r of uo[p 4] 


x=[5 i] v«@[9 ol: 


The set of stabilizing controllers is given by 


so=-(fr fem ED) (a tem aD): 


have particular solutions 


and 
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where W, varies over Rs(z), or by 


O74 1 0 1 1]. fet! -2-1422-2 - 
40=—([) ol-[- ee) ((o al+[o A me) 


where Wr varies over Rs(z) as well. 

It is clear that the two parameterizations of Sz are equivalent. To each controller S there is a unique 
parameter Wy such that S; = —(X;, + W_B,)7! (Y;, — W_Arz) as well as a unique parameter Wa such that 
Sp = —(Yp—ArWr)(Xp+Br Wr) !, and these two are related by 


Wr — Wi =X1Yr—Y,Xp- 


It is easy to see that the transfer matrix of the closed-loop system is affine in the free parameter W,, or 
Wr. Indeed, 
Yi} | BrX,+WBr) = —Br(¥;, — WLAL) uy 
ye IT—Ar(X,+ W1BL) —Ar(Y, — WLAL) u2 


or alternatively 


NY ‘s | (Xp + BrWr)Br ae | | 
y2 —(Yp—-ArWr)B, —(Y¥p—ArWr)AL u2 | 


Thus, control synthesis problems beyond stabilization can be handled by determining the parameters 
W_ or Wras described for SISO systems. 
Let us consider the disturbance attenuation problem for the discrete-time plant 


= Doe = 
Suey | 2 0 ar? |e Sele ma |e 


z—-—l z—l1 z—-—l1 


where the disturbance d is assumed to be an arbitrary [,) sequence. We seek to find a BIBO stabilizing 
controller that minimizes the maximum amplitude of the plant output y. 
We write 


Siu(z) = Ay (z)Bi(z) = Br(z)AR' (z)s 
Sia(z) = Az (z)Ci(z); 


where 


= 9,71 
Aue) =| ak ae=[, ba |: cta= (2, | 


= es -1 
Ane)=| 7 mel Bale) = [* se |: 


The set of BIBO stabilizing controllers has been found to have the transfer matrix 


0 1 1 0 Pa) fer! S28 oe? a 
wo. Sale) Qe 9)" 


where Wr varies over Rs(z). 
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The disturbance-output transfer matrix equals 


G(z) = (I — SiuS2)7*Sia 
= (Xp + BrWr) !Cy. 


When 


Wi | 
Wr= ; 
* Ee Wo 


one obtains the expression 


zt 4277 Wy — (27? — 2274) Wop 
G(z) = 3 3 ; 
Zo +z “Wo 


The 1-norm of an m x n matrix G(z) with entries 


CO 
Gy) = >° gyez* 


k=0 
is defined by 
n 
|Gll; = max Galli, 
i=1,..,.m 

j=l 
where 

n 

Gili = >— leial- 
k=0 
In our case 


Gl], = max (Jz) +27? Wy — (2? = 2277) Wall, Ile +27? Wool), 
and it is clear by inspection that ||G||, attains its minimum for W12(z) = 0, W22(z) = 0 and 


min ||G||; = max(1, 1) = 1. 
Wr 


The corresponding optimal BIBO stabilizing controllers are 


-—Wi 1 1+z271Wiy—(z-!-2277)Wa 1 p 
S2(z) = — - La 
Wi1—-(1—z*)W2, 0 z Wo 1 


for any functions Wj; and W2, in Rs(z). The one of least McMillan degree reads 
0 -!il 
S(z) = I 0 . 


21.12 Extensions 


The factorization approach presented here for linear time-invariant systems with rational transfer matrices 
can be generalized to extend the scope of the theory to include distributed-parameter systems, time- 
varying systems, and even nonlinear systems. 
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The transfer matrices of distributed-parameter systems are no longer rational and coprime factoriza- 
tions cannot be assumed a priori to exist. The coefficients of time-varying systems are functions of time, 
and the operations of multiplication and differentiation do not commute. In nonlinear systems, transfer 
matrices are replaced by input-output maps. Suitable factorizations of these maps may not exist and, if 
they do, they are not commutative in general. 

For many systems of physical and engineering interest, these difficulties can be circumvented and the 
algebraic factorization approach carries over with suitable modifications. 
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22.1.1 Quantitative Feedback Theory 


Quantitative feedback theory (QFT)* is a very powerful design technique for the achievement of assigned 
performance tolerances over specified ranges of structured plant parameter uncertainties without and with 


* 


The original version of this material was first published by the Advisory Group for Aerospace Research and Development, 


North Atlantic Treaty Organization (AGARD/NATO) in Lecture Series LS-191 “on Linear Dynamics and Chaos” in 


June 1993. 
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control effector failures [9]. It is a frequency domain design technique utilizing the Nichols chart (NC) 
to achieve a desired robust design over the specified region of plant parameter uncertainty. This chapter 
presents an introduction to QFT analog and discrete design techniques for both multiple-input single- 
output (MISO) [1,5,13] and multiple-input multiple-output (MIMO) [3,4,6,7,10-12] control systems. 
QFT computer-aided design (CAD) packages are readily available to expedite the design process. The 
purposes of this chapter are (1) to provide a basic understanding of QFT; (2) to provide the minimum 
amount of mathematics necessary to achieve this understanding; (3) to discuss the basic design steps; and 
(4) to present a practical example. 


22.1.2 Why Feedback? 


For the answer to the question of "Why do you need QFT?" consider the following system. The plant P 
responds to the input r(t) with the output y(t) in the face of disturbances dj (t) and d2(t) (see Figure 22.1). 
If it is desired to achieve a specified system transfer function T(s)[= Y(s)/R(s)], then it is necessary to 
insert a prefilter whose transfer function is T(s)/P(s), as shown in Figure 22.2. This compensated system 
produces the desired output as long as the plant does not change and there are no disturbances. This 
type of system is sensitive to changes in the plant (or uncertainty in the plant), and the disturbances are 
reflected directly into the output. Thus, it is necessary to feed back the information in the output in order 
to reduce the output sensitivity to parameter variation and to attenuate the effect of disturbances on the 
plant output. 
In designing a feedback control system, it is desired to utilize a technique that: 


« Addresses all known plant variations up front 
¢ Incorporates information on the desired output tolerances 
¢ Maintains reasonably low loop gain (reduce the “cost of feedback”) 


This last item is important in order to avoid the problems associated with high loop gains such as sensor 
noise amplification, saturation, and high-frequency uncertainties. 


22.1.3 What Can QFT Do? 


Assume that the characteristics of a plant that is to be controlled over a specified region of operation vary; 
that is, a plant with structured parameter uncertainty. This plant parameter uncertainty may be described 
by the Bode plots of Figure 22.3. This figure represents the range of variation of plant magnitude (dB) and 
phase over a specified frequency range. The bounds of this variation, for this example, can be described 
by six linear time-invariant (LTI) plant transfer functions. By the application of QFT for a MISO control 
system containing this plant, a single compensator and a prefilter may be designed to achieve a specified 


robust design. 
P 


FIGURE 22.1 An open-loop system (basic plant). 
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FIGURE 22.2 A compensated open-loop system. 
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FIGURE 22.3 The Bode plots of six LTI plants that represent the range of the plant’s parameter uncertainty. 


22.1.4 Benefits of QFT 


The benefits of QFT may be summarized as follows: 


The result is a robust design that is insensitive to plant variation 

There is one design for the full envelope (no need to verify plants inside templates) 

Any design limitations are apparent up front 

In comparison to other multivariable design techniques there is less development time for a full 
envelope design 

One can determine what specifications are achievable early in the design process 

One can redesign quickly for changes in the specifications 

The structure of compensator (controller) is determined up front 


22.2 The MISO Analog Control System [1] 


22.2.1 Introduction 


The mathematical proof that an m x m feedback control system can be represented by m equivalent MISO 
feedback control systems is given in Section 22.4.2. A 3 x 3 MIMO control system can be represented 
by the m? MISO equivalent loops shown in Figure 22.4. Thus, this and the next section present an 
introduction to the QFT technique by considering only a MISO feedback control system. 


22.2.2 MISO System 


The overview of the MISO QFT design technique is presented in terms of the minimum-phase (m.p.) LTI 
MISO system of Figure 22.5. The control ratios for tracking (D = 0) and for disturbance rejection (R = 0) 
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FIGURE 22.4 m? MISO equivalent of a 3 x 3 MIMO feedback control system. 


are, respectively, 
F(s)G(s)P(s) — F(s)L(s) 
~ 14G(s)P(s) 1+ L(s) 
P(s) P(s) 
~ 14+G(s)P(s) 1+ L(s) 
The design objective is to design the prefilter F(s) and the compensator G(s) so the specified robust 


design is achieved for the given region of plant parameter uncertainty. The design procedure to accomplish 
this objective is as follows: 


R (22.1) 


Tp (22.2) 


Step 1: Synthesize the desired tracking model. 

Step 2: Synthesize the desired disturbance model. 

Step 3: Specify the J LTI plant models that define the boundary of the region of plant parameter 
uncertainty. 

Step 4: Obtain plant templates, at specified frequencies, that pictorially describe the region of plant 
parameter uncertainty on the NC. 

Step 5: Select the nominal plant transfer function P,(s). 

Step 6: Determine the stability contour (U-contour) on the NC. 

Steps 7-9: Determine the disturbance, tracking, and optimal bounds on the NC. 

Step 10: Synthesize the nominal loop transmission function Lo(s) = G(s)Po(s) that satisfies all the 
bounds and the stability contour. 

Step 11: Based upon Steps 1 to 10 synthesize the prefilter F(s). 

Step 12: Simulate the system in order to obtain the time response data for each of the J plants. 


The following sections illustrate this design procedure. 


f 
lee Ob >C 
+ 


FIGURE 22.5 A MISO plant. 
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FIGURE 22.6 Desired response characteristic: (a) thumbprint specifications; (b) Bode plots of Tp. 


22.2.3 Synthesize Tracking Models 


The tracking thumbprint specifications, based upon satisfying some or all of the step-forcing function 
figures of merit for underdamped (Mp, tp, ts, tr, Km) and overdamped (t;, t;; Km) responses, respectively, 
for a simple-second order system, are depicted in Figure 22.6a. The Bode plots corresponding to the time 
responses y(t)y (Equation 22.3) and y(t), (Equation 22.4) in Figure 22.6b represent the upper bound 
By and lower bound Br, respectively, of the thumbprint specifications; i.e., an acceptable response y(t) 
must lie between these bounds. Note that for m.p. plants, only the tolerance on |Tr(jw;)| need be satisfied 
for a satisfactory design. For nonminimum-phase (n.m.p.) plants, tolerances on ZTp(jwj;) must also be 
specified and satisfied in the design process [4,5]. It is desirable to synthesize the tracking control ratios 


(«2 /a)(s + a) 
= 22.3 
Be ke 2Cans + 2 (23) 
K 
fe (22.4) 


(s — 61)(s — o2)(s — 63) 


corresponding to the upper and lower bounds Tr, and Tr,, respectively, so that 8e(jw;) = Bu — Br 
increases as w; increases above the 0-dB crossing frequency of Tr,,. This characteristic of Sp simplifies 
the process of synthesizing L,(s) = G(s)Po(s). This synthesis process requires the determination of the 
tracking bounds Br(jw;) that are obtained based upon 8r(jw;). The achievement of the desired perfor- 
mance specification is based upon the frequency bandwidth (BW), 0 < w < w , which is determined by 
the intersection of the —12-dB line and the By curve in Figure 22.6b. 


22.2.4 Disturbance Model 


The simplest disturbance control ratio model specification is |Tp(jw)| = |¥(jw)/D(jw)| < ap a constant 
[the maximum magnitude of the output based upon a unit-step disturbance input (d; of Figure 22.1)]. 
Thus, the frequency domain disturbance specification is log magnitude (Lm) Tp(jm) < Lm ap» over 
the desired specified BW 0 < w < wy, as defined in Figure 22.6b. Thus, the disturbance specification is 
represented by only an upper bound on the NC over the specified BW. 


22.2.5 J LTI Plant Models 


The simple plant 
Ka 
Ps) = 22.5 
ifs) s(s+a) ee) 
where Ke{1, 10} and ae{1, 10}, is used to illustrate the MISO QFT design procedure. The region of plant 
parameter uncertainty is illustrated by Figure 22.7. This region of uncertainty may be described by J LTI 
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FIGURE 22.7 Region of plant uncertainty characterizing Equation 22.5. 


plants, where j = 1,2,...,J. These plants lie on the boundary of this region of uncertainty; that is, the 
boundary points 1 to 6 are utilized to obtain six LTI plant models that adequately define the region of 
plant parameter uncertainty. 


22.2.6 Plant Templates of P;(s), 3 P( jw;) 
With L = GP, Equation 22.1 yields 


L 
LmTrp = LmF —Lm (22.6) 
1+L 
The change in Tr due to the uncertainty in P, since F is LTI, is 


L 

A(LmTr) = LmTp —LmF = Lm Fea (22.7) 
By the proper design of L, = GP, and F, this change in Tp is restricted so that the actual value of Lm Tp 
always lies between By and By of Figure 22.6b. The first step in synthesizing an Ly is to make NC templates 
that characterize the variation of the plant uncertainty (see Figure 22.8), as described by j = 1,2,...,J 
plant transfer functions, for various values of w; over a specified frequency range. The boundary of the 
plant template can be obtained by mapping the boundary of the plant parameter uncertainty region, Lm 
P;(ja;) vs. ZP;(ja;), as shown on the NC in Figure 22.8. A curve is drawn through the points 1, 2, 3, 4, 5, 
and 6 where the shaded area is labeled SP(j1), which can be represented by a plastic template. Templates 
for other values of w; are obtained in a similar manner. A characteristic of these templates is that, starting 


FIGURE 22.8 The template SP(j1) characterizing Equation 22.5. 
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from a “low value” of «;, the templates widen (angular width becomes larger) for increasing values of w;, 
then, as w; takes on larger values and approaches infinity, they become narrower and eventually approach 
a straight line of height V dB (see Equation 22.9). 


22.2.7 Nominal Plant 


While any plant case can be chosen, select whenever possible a plant whose NC template point is always 
at the lower left corner for all frequencies for which the templates are obtained. 


22.2.8 U-Contour (Stability Bound) 


The specifications on system performance in the frequency domain (see Figure 22.6b) identify a minimum 
damping ratio ¢ for the dominant roots of the closed-loop system, which becomes a bound on the value 
Mp © Mm. On the NC this bound on Mm = M_ (see Figures 22.6b and 22.9) establishes a region that must 
not be penetrated by the templates and the loop transmission function L( jw) for all w. The boundary 
of this region is referred to as the universal high-frequency boundary (UHFB) or stability bound—the 
U-contour, because this becomes the dominating constraint on L( jw). Therefore, in Figure 22.9 the top 
portion, efa, of the M;, contour becomes part of the U-contour. For a large problem class, as w — oo, the 
limiting value of the plant transfer function approaches 


K 

lim [P(jo)] = (22.8) 
@—>0o oO 

where > represents the excess of poles over zeros of P(s). The plant template for this problem class 


approaches a vertical line of length equal to 


A lim ' [LmPmax — LmP min] = LmKmax — LmKpin = VdB (22.9) 


(woo 


If the nominal plant is chosen at K = Kin, then measuring VdB down from the bottom portion age of 
the constraint My, gives the bcd portion of the U-contour abcdefa of Figure 22.9. The remaining portions, 
ab and de, of the stability contour are determined during the process of determining the tracking bounds. 


22.2.9 Optimal Bounds Bo( jw;) on Lo| j;) 
The determination of the tracking Ba(jw;) and the disturbance Bp(jw;) bounds are required in order to 


yield the optimal bounds Bo(jw;) on Lo(jw;). 


22.2.9.1 Tracking Bounds 


The solution for Br(jw;) requires that the condition (actual) ATp(jw;) < 8r(jw;) dB (see Figure 22.6b) 
must be satisfied. Thus, it is necessary to determine the resulting constraint, or bound Ba( joj), on L( jai). 


B,-boundary 


U-contour LS 


FIGURE 22.9 U-contour construction. 
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The procedure is to pick a nominal plant P,(s) and to derive tracking bounds on the NC, at specified 
values of frequency, by use of templates or a CAD package. That is, along a phase angle grid line on the 
NC, move the nominal point on the template 3P(jw;) up or down, without rotating the template, until it 
is tangent to two M-contours whose difference in M values is essentially equal to 8p. When this condition 
has been achieved, the location of the nominal point on the template becomes a point on the tracking 
bound Br(jw;) on the NC. This procedure is repeated on sufficient angle grid lines on the NC to provide 
sufficient points to draw Br( jw;) and for all values of frequency for which templates have been obtained. 
In general, the templates are moved from right to left starting from a phase angle grid line to the right 
of the M;, contour. When the templates become tangent to the M; contour, the nominal point on the 
templates yields points on the ab and de portions of the stability contour. For m.p. systems, the condition 
ATr(jwi) < dr(joj) requires that the synthesized loop transmission must satisfy the requirement that 
Lm L( j«;) is on or above the corresponding tracking bound Lm Br(jw)). 


22.2.9.2 Disturbance Bounds 


The general procedure for determining disturbance bounds for the MISO control system of Figure 22.5 
is outlined as follows, but more details are given in [1]. From Equation 22.2 the following equation is 


obtained: 
Po Po 


Tp = ———X“-— = — 
> (Po/P) +o W 


(22.10) 
where W = (P,/P) +L . From Equation 22.10, setting LmTp = 8p = Lmas, the following relationship 
is obtained: 

LmW = LmP, — 8p (22.11) 


For each value of frequency for which the NC templates are obtained, the magnitude of |W(jw)| 
is obtained from Equation 22.11. This magnitude, in conjunction with the equation W(ja;) = 
[Po( jai)/P(jai)], is utilized to obtain a graphical solution for Bp(jw;) as shown in Figure 22.10. Note 
that in this figure the template is plotted in rectangular or polar coordinates. 


22.2.9.3 Optimal Bounds 


For the case shown in Figure 22.11, Bo(jw;) is composed of those portions of each respective bound 
Br( joj) and Bp(jw;) that have the largest dB values. The synthesized Lo(jw;) must lie on or just above 
the bound B,( ja;) of Figure 22.11. 


22.2.10 Synthesizing (or Loop-Shaping) L,(s) and F(s) 


The shaping of Lo( jw) is shown by the dashed curve in Figure 22.11. A point such as Lm L,(j2) must be 
on or above B,( j2). Further, in order to satisfy the specifications, Lo( jw) cannot violate the U-contour. In 
this example, a reasonable Lo( jw) closely follows the U-contour up to w = 40 rad/s and must stay below 
it above w = 40, as shown in Figure 22.11. It also must be at least a Type 1 L,(s) transfer function (one or 


| W(ja)| 


> —180° 


FIGURE 22.10 Graphical evaluation of Bp( jw). 
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FIGURE 22.11 Bounds B,( ja;) and loop-shaping. 


more poles at the origin) for tracking a step-forcing function with zero steady-state error [1]. Synthesizing 
a rational function L,(s) that satisfies the above specification involves building up the function 


Lo( jo) = Lok (Jo) = Pol jo) Mpg [Kk Gx(jo)] (22.12) 


where for k = 0, Gp = 120° and K = [ [py Kg. In order to minimize the order of the compensator, a good 
starting point for “building up” the loop transmission function is to assume initially that Loo (jw) = Po( ja) 
as indicated in Equation 22.13. L,(jw) is built up term-by-term or by a CAD loop-shaping routine [8], 
in order (1) that the point L,(jw;) lies on or above the corresponding optimal bound Bo( jw;), (2) that it 
passes close to the trough of the low frequency bounds, for achieving minimal gain, and (3) to stay just 
outside the U-contour in the NC of Figure 22.11. The design of a proper L,(s) guarantees only that the 
variation in | Tp(joj)| is less than or equal to that allowed; ie., 8a(jw;). The purpose of the prefilter F(s) 
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is to position Lm [T(jw)] within the frequency domain specifications; i.e., that it always lies between By 
and By; (see Figure 22.6b) for all J plants. The method for determining F(s) is discussed in the next section. 
Once a satisfactory Lo(s) is achieved, then the compensator is given by G(s) = Lo(s)/Po(s). Note that for 
this example L,(jw) slightly intersects the U-contour at frequencies above w);. Because of the inherent 
overdesign feature of the QFT technique, as a first trial design no effort is made to fine-tune the synthesis 
of L,(s). If the simulation results are not satisfactory, then a fine tuning of the design can be made. The 
available CAD packages simplify and expedite this fine tuning. 


22.2.11 Prefilter Design [1,2,4,5] 


Design of a proper L,(s) guarantees only that the variation in | Tr(jo)| is less than or equal to that allowed; 
ie, [LmTp(jo)] < 8e(jw) The purpose of the prefilter F(s) is to position 


L(jw) 


LmT(jw) = Lm 


within the frequency domain specifications. A method for determining the bounds on F(s) is as follows: 


Step 1: Place the nominal point of the w; plant template on the Lo(ja;) point on the Lo( jw) curve on 
the NC (see Figure 22.12). 

Step 2: Traversing the template, determine the M-contours that yield the maximum LmTmax and the 
minimum LmT min values of Equation 22.13. 

Step 3: Based upon obtaining sufficient data points, by repeating Step 2 within the desired frequency 
bandwidth for various values of w;, and in conjunction with the data used to obtain Figure 22.6b 
the plots of Figure 22.13 are obtained. 

Step 4: Utilizing Figure 22.13, the straight-line Bode technique, and the condition 


lim F(s)=1 (22.14) 


for a step-forcing function, an F(s) is synthesized that lies within the upper and lower plots in 
Figure 22.13. 
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FIGURE 22.12 Prefilter determination. 
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FIGURE 22.13 Frequency bounds on F(s). 


22.2.12 Simulation 


The “goodness” of the synthesized L,(s) and F(s) is determined by simulating the QFT-designed control 
system for all J plants. MISO QFT CAD packages, as discussed in Section 22.2.4, are available to expedite 
this simulation phase of the complete design process. 


22.2.13 MISO QFT CAD Packages 


The first usable MISO QFT CAD package was developed in 1986 for the analog design and in 1991 for the 
discrete design at the Air Force Institute of Technology (AFIT). These CAD packages have been a catalyst 
in assisting the newcomer to QFT to understand the fundamentals of this powerful design technique. 
The QFT CAD package illustrated in Appendix B can be used for both MISO and MIMO control system 
designs. 


22.2.13.1 MISO QFT CAD 


The flowchart of the MISO QFT CAD options in the AFIT package called TOTAL-P.C is shown in 
Appendix A. Those desiring a copy of this package can contact Professor C. H. Houpis, AFIT/ENG, 
Wright-Patterson AFB, OH 45433. This package has been designed as an educational tool. The QFT CAD 
package of Appendix B can also be used for the design of a MISO control system. 


22.2.13.2 MISO QFT PC CAD 


Dr. Yossi Chait, University of Massachusetts, and Dr. Oded Yaniv, Tel-Aviv University, Israel, have devel- 
oped a MISO QFT PC CAD package for both analog and discrete system design available in MATLAB®. 


22.3 The MISO Discrete Control System [13] 


22.3.1 Introduction 


The bilinear transformation, z-domain to the w’-domain and vice-versa, is utilized in order to accomplish 
the QFT design for both MISO and MIMO sampled-data (discrete) control system design in the w’- 
domain. This transformation enables the MISO QFT analog design technique to be readily used, with 
minor exceptions, to perform the QFT design for the controller G(w’). If the w’-domain simulations 
satisfy the desired performance specification, then by use of the bilinear transformation the z-domain 
controller G(z) is obtained. With this z-domain controller, a discrete-time domain simulation is obtained 
to verify the “goodness” of the design. The QFT technique requires the determination of the minimum 
sampling frequency (w;)min BW that is needed for a satisfactory design [13,14]. The larger the plant 
uncertainty and the narrower the system performance tolerances, the larger must be the value of (#5) min. 
Henceforth, the prime is omitted from w’; whenever the symbol w is used it is to be interpreted as w’. 
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FIGURE 22.14 A MISO sampled-data control system. 


22.3.2 The MISO Sampled-Data Control System 


Figure 22.14 represents the MISO discrete control system, having plant uncertainty, that is to be designed 
by the QFT technique. The equations that describe this system are as follows: 


P.(z) = GP(z) = (1—27))Z 7] = (1—z7!)P,(z) (22.15) 
L(z) = GyoP(z)G\(z), Pe = — P.(z)=Z 72] =Z[P.] (22.16) 
Dis) =~ Pals) = P(S)D() 

P.(z) = Z[P(s)D(s)] = PD(z) (22.17) 

F(z)L(z) PD(z) 
TR Tay 8 = TL) (218) 
_ [L@F) PD(z) _ 
Y(z)= | oe] R(z)+ ai Yr(z) + Yp(z) 
= Tr(z)R(z) + Yp(z) (22.19) 


22.3.3 w-Domain 


The pertinent s-, z-, and w-plane relationships are as follows: 


5s HOEY wT 
w=(—) <2, —<0.297 (22.20) 
2; 2 
s=o+ja, (a) 
2 z—1 
= iy = b 22.21 
w=utjv (7)(S] (b) (22.21) 
Tw+2 2 T 
Z= es , v=—tan S aay (ee tan ae (22.22) 
—Tw+2 T 2 Ws Ws 
®,=2n/T, c=! ZT =|z|ZwT (22.23) 


22.3.4 Assumptions 


For this chapter, the following assumptions are made: 


¢ Minimum-phase (m.p.) stable plants 
¢ The analog design models, Equations 22.3 and 22.4, yield the desired time response characteristics 
for the discrete-time system 
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« The sampling time T is small enough so that over the BW, 0 < wm < wy, Equation 22.23 is valid 
permit- ting the approximation s © w and, in turn, 


Tr(w) © [Tr(s)]—w (22.24) 


Both the upper and lower bound w-domain tracking models are obtained in this manner. The distur- 
bance specification is the same as for the analog case. 


22.3.5 Nonminimum Phase L,(w) 


It is important to note that in the w domain any practical L(w) is n.m.p. containing a zero at 2/T (the 
sampling zero). This result is due to the fact that any practical L(z) has an excess of at least one pole over 
zeros. Thus, the design technique for a stable uncertain plant is modified [14] to incorporate the allpass 
filter (apf) 


w—(2/T) ' SAS 
A(w) = ———— =-A = 22.25 
w= FO =~ Lam-ew wae) 
as follows. Let the nominal loop transmission be defined as 
Lo = —Lino(w)A(w) = Lino(w)A’(w) (22.26) 
From Equation 22.26 it is seen that 
ZL mo( jv) = ZLo( jv) — ZA'(jv) (22.27) 
where 
ips _, VT 
—ZA'(jv) =2 tan oF >0 (22.28) 


An analysis of Equations 22.26 through 22.28 reveals that the bounds B/(jv;) on Lo(jv) become the 
bounds Byo(jvj) on Lmo( jv) by shifting, over the desired BW, Bi (jv;) positively (to the right on the NC) 
by the angle 7A’(jv;), as shown in Figure 22.15. The U-contour (Bj,) must also be shifted to the right by 
the same amount, at the specified frequencies v;, to obtain the shifted U-contour B,(jv;). The contour 
B, is shifted to the right until it reaches the vertical line ZL no, (jvK) = 0°. The value of vx, which is a 
function of w; and the phase margin angle y as shown in Figure 22.15, [13] is given by 


VET 
2tan7} (=) = 180° —y (22.29) 


It should be mentioned that loop-shaping or synthesizing Lo(w) can be done directly without the use 
of an apf. 


22.3.6 Plant Templates 3 P(jv1) 


The plant templates in the w-domain have the same characteristic as those for the analog case (see 
Section 22.2.4) for the frequency range 0 < w; < w,/2 as shown in Figure 22.16a. In the frequency range 
w;/2 <j < oo, the w-domain templates widen once again, then eventually approach a vertical line as 
shown in Figure 22.16b. 


22.3.7 Synthesizing Lyo(w) 


The frequency spectrum can be divided into four general regions for the purpose of synthesizing (loop- 
shaping) an Lino(w) that satisfies the desired system performance specifications for the plant having plant 
parameter uncertainty. These four regions are 
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FIGURE 22.15 The shifted bounds on the NC. Note: Curves drawn approximately to scale. 
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FIGURE 22.16 w-domain plant templates. 
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Region 1: For the frequency range where Equation 22.20 is essentially satisfied, use the analog templates; 
ie, SP(jw;) ¥ SP(jvj). The w-domain tracking, disturbance, and optimal bounds and the U- 
contour are essentially the same as those for the analog system. The templates are used to obtain 
these bounds on the NC in the same manner as for the analog system. 

Region 2: For the frequency range vo.25 < vj < vq, where w; < 0.25m;, use the w-domain templates. 
These templates are used to obtain all three types of bounds, in the same manner as for the analog 
system, in this region; and the corresponding B,, (jv;) contours are also obtained. Depending on 
the value of T, v;, may be less than v9.25. 

Region 3: For the frequency range vj, < v; < vx, for the specified value of w,, only the B), contours are 
plotted. 

Region 4: For the frequency range v_, > vx, use the w-domain templates. Since the templates 3P.(jv;) 
broaden out again for v; > vx, as shown in Figure 22.16, it is necessary to obtain the more stringent 
(stability) bounds Bs shown in Figure 22.17. The templates are used only to determine the stability 
bounds Bs. 


The synthesis (or loop-shaping) of L,9(w) involves the synthesis of the following function: 


Limo(jv) = Peo( jv) ic [KiGx(jv)] (22.30) 


where the nominal plant P.o(w) is the plant from the J plants that has the smallest dB value and the largest 
(most negative) phase lag characteristic. The final synthesized L»,.(w) function must be one that satisfies 
the following conditions: 


1. In Regions 1 and 2 the point on the NC that represents the dB value and phase angle of Lino(jv;) 
must be such that it lies on or above the corresponding Byyo(jvj) bound (see Figure 22.15). 

2. The values of Equation 22.30 for the frequency range of region 3 must lie to the right of or just 
below the corresponding B), contour (see Figure 22.15). 

3. The value of Equation 22.30 for the frequency range of region 4 must lie below the Bs contour for 
negative phase angles on the NC (see condition 4 next). 

4. In utilizing the bilinear transformation of Equation 22.21, the w-domain transfer functions are all 
equal order over equal order. 

5. The Nyquist stability criterion dictates that the L,,,(jv) plot is on the “right side” or the “bottom 
right side” of the B,(jv;) contours for the frequency range of 0 < v; < vx. Ithas been shown that [3] 
a. Lmo(jv) must reach the right-hand bottom of By(jvx), (i.e., approximately point K in 

Figure 22.17) at a value of v < vx. 
b. ZL no(jvx) < 0° in order that there exists a practical Lm that satisfies the bounds B( jv) 
and provides the required stability. 

6. For the situation where one or more of the J LT] plants that represent the uncertain plant parameter 
characteristics represent unstable plants and one of these unstable plants is selected as the nominal 
plant, then the apf to be used in the QFT design must include all right half-plane (RHP) zeros of 
Pz,. This situation is not discussed in this chapter. Note: For experienced QFT control system 
designers, Ly(v) can be synthesized without the use of apf. This approach also is not covered in 
this chapter. 


The synthesized Lyo(w) obtained following the guidelines of this section is shown in Figure 22.17. 


22.3.8 Prefilter Design 


The procedure for synthesizing F(w) is the same as for the analog case (see Section 22.2.11) over the 
frequency range 0 < vj < vj. In order to satisfy condition 4 of Section 22.3.7, a nondominating zero or 
zeros (“far left” in the w-plane) are inserted so that the final synthesized F(w) is equal order over equal 
order. 
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FIGURE 22.17 A satisfactory design: Lino( jv) at ws = 240. 
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22.3.9 w-Domain Simulation 


The “goodness” of the synthesized Ly9(w) [or Lo(w)] and F(w) is determined by first simulating the QFT 
w-domain designed control system for all J plants in the w-domain (an “analog” time domain simulation). 
See Section 22.2.13 for MISO QFT CAD packages that expedite this simulation. 


22.3.10 z-Domain 


The two tests of the “goodness” of the w-domain QFT-designed system is a discrete-time domain simula- 
tion of the system shown in Figure 22.14. To accomplish this simulation, the w-domain transfer functions 
G(w) and F(w) are transformed to the z-domain by use of the bilinear transformation of Equation 22.21. 
This transformation is utilized since the degree of the numerator and denominator polynomials of these 
functions are equal and the controller and prefilter do not contain a zero-order-hold device. 


22.3.10.1 Comparison of the Controller’s w- and z-Domain Bode Plots 


Depending on the value of the sampling time T, warping may be sufficient to alter the loop-shaping 
characteristics of the controller when it is transformed from the w-domain into the z-domain. For the 
warping effect to be minimal, the Bode plots (magnitude and angle) of the w- and z-domain controllers 
must essentially lie on top of one another within the frequency range 0 < w < [(2/3)(w;/2)]. If the 
warping is negligible, then a discrete-time simulation can proceed. If not, a smaller value of sampling time 
needs to be selected. 


22.3.10.2 Accuracy 


The CAD packages that are available to the designer determines the degree of accuracy of the calculations 
and simulations. The smaller the value of T, the greater the degree of accuracy that is required to be 
maintained. The accuracy can be enhanced by implementing G(z) and F(z) as a set of g and f cascaded 
transfer functions, respectively; that is, 


G(z) = Gi(z)Ga(z) ++ - Ge(z), 
F(z) = Fy(z)Fo(z)- ++ F(z) (22.31) 


22.3.10.3 Analysis of Characteristic Equation Q;(z) 


Depending on the value of T and the plant parameter uncertainty, the pole-zero configuration in the 
vicinity of the —1+j0 point in the z-plane for one or more of the J LTI plants can result in an unstable 
discrete-time response. Thus, before proceeding with a discrete-time domain simulation, an analysis of 
the characteristic equation Q;(z) for all J LTI plants must be made. Ifan unstable system exists, an analysis 
of Q;(z) and the corresponding root locus may reveal that a slight relocation of one or more controller 
poles in the vicinity of the —1+j0 point toward the origin may ensure a stable system for all J plants 
without essentially affecting the desired loop-shaping characteristic of G(z). 


22.3.10.4 Simulation and CAD Packages 


With the “design checks” of Sections 22.3.10.1 through 22.3.10.3 satisfied, then a discrete-time simulation 
is performed to verify that the desired performance specifications have been achieved. In order to enhance 
the MISO QFT discrete control system design procedure that is presented in this chapter, the CAD flow 
chart of Section 22.2.13.1 is shown in Appendix B. 
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22.4 MIMO Systems 


22.4.1 Introduction 


Figure 22.18 represents an m x m MIMO closed-loop system in which F,G, and P are each mx m 
matrices, and P = {P} is a set of J matrices due to plant parameter uncertainty. There are m* closed- 
loop system transfer functions (transmissions) fj; contained within its system transmission matrix (i.e., 
T = {tj}) relating the outputs y; to the inputs 7; (e.g., yj = trj). These relationships hold for both the 
s- and w-domain analysis of a MIMO system. In a quantitative problem statement there are tolerance 
bounds on each tj, giving a set of m? acceptable regions 1; that are to be specified in the design; thus, 
tj€t and 3 = {tj}. From Figure 22.18 the system control ratio relating r to y is: 


T =[I1+PG]"!PGF (22.32) 


The tj expressions derived from this expression are very complex and not suitable for analysis. The QFT 
design procedure systematizes and simplifies the manner of achieving a satisfactory system design for 
the entire range of plant uncertainty. In order to readily apply the QFT technique, another mathematical 
system description is presented in the next section. The material presented in this chapter pertains to 
both the s- and w-domain analysis of MIMO systems [2,12,16,17]. 


22.4.2 Derivation of m? MISO System Equivalents 


The G, F, P and P~! matrices are defined as follows: 


go 0 --. 0 fir fiz -:+ fim 
0 gw -:: O fu fez s+ fam 

G=|/.., ; . F=| , . : : (22.33) 
0 O +++ gm ml fm2*** fmm 
Pi Piz t+ Pim Pip Pig =? Pip 
por poz +++ Pom | Pn Po om 

P=| , F F . PY=| . : ; . (22.34) 
Pmi Pm2 +++ Pmm Pig Pinas 8? Din 


Although only a diagonal G matrix is considered, the use of a nondiagonal G matrix may allow the 
designer more design flexibility [2]. The m? effective plant transfer functions are based upon defining: 
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FIGURE 22.18 A 3 x 3 MIMO feedback control system. 


(22.35) 
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There is a requirement that det.P be m.p. The Q matrix is then formed as: 


1 1 1 
Pio Pp Pim 
qr fiz *** Gm a hs sie 
Q = 21 2 ane q2m — Pr Px Pom (22.36) 
Gm m2 “++ Amm 
1 1 1 
Pim Pm2 Pinm 
The matrix P~! is partitioned to the form: 
—1 * 1 
P} = [Pi]= ate A+B (22.37) 
y 


where A is the diagonal part of P~! and B is the balance of P~!; thus hi = 1/qi = pj; bi = 0, and 
by = 1/qi = Pi for i ~ j. Premultiplying Equation 22.32 by [I + PG] yields 


[I+ PG]T = PGF > [P-!+ G]T =GF (22.38) 


where P is nonsingular. Using Equation 22.37 with G diagonal, Equation 22.38 can be rearranged to the 
form: 
T =[A+G]_'[GF - BT] (22.39) 


Equation 22.39 is used to define the desired fixed-point mapping where each of the m? matrix elements 
on the right-hand side of this equation can be interpreted mathematically as representing a MISO system. 
Proof of the fact that design of each MISO system yields a satisfactory MIMO design is based on the 
Schauder fixed-point theorem [7]. This theorem is described, based upon a unit impulse input, by defining 
a mapping 

Y(Ti) =[A + G]'[GF — BT;] = T; (22.40) 


where each member of T is from the acceptable set 3. If this mapping has a fixed point [i.e., TeS such that 
Y(T;) = T;], then this T is a solution of Equation 22.39. The y;, output obtained from Equation 22.40, 
for the 3 x 3 case, is given by: 


qt t] t3] ) 
= + (22.41) 
1+g1411 [svn (2 13 


Based upon the derivation of all the y;; expressions from Equation 22.40 yields the four effective MISO 


Vi 


loops (in the boxed area) in Figure 22.4, resulting from a 2 x 2 system and the nine effective MISO loops 
resulting from a3 x 3 system [4]. The control ratios for the desired tracking inputs r; by the corresponding 
outputs y; for each feedback loop of Equation 22.40 have the form 


Vii = Wii (Vij + cy) (22.42) 
where wij = qii/(1 + gigi) and vij = gif. The interaction between the loops has the form 
g=e>) || 2) ease (22.43) 
kAi | dik 


and appears as a “cross-coupling effect” input in each of the feedback loops. Thus, Equation 22.42 
represents the control ratio of the ith MISO system. The transfer function w;;vj relates the “desired” ith 
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output to the jth input 7;, and the transfer function w,jcj relates the ith output to the jth cross-coupling 
effect input cj. The outputs given in Equation 22.42 can thus be expressed as 


Vii = iri + Vey = Ini + Vey (22.44) 
or, based on a unit impulse input, 
tig = try + bey (22.45) 
where 
try =Vri = Wiig — bey = Vow = Wii ij (22.46) 


and where now the upper bound, in the low-frequency range (0 < w < wy), is expressed as bi. Thus, 
Toy = by — bij (22.47) 


represents the maximum portion of bj allocated toward the cross-coupling effect rejection, and bij 
represents the upper bound for the tracking portion, respectively, of tj. For each MISO system there is a 
cross-coupling effect input that is a function of all the other loop outputs. The object of the design is to 
have each loop track its desired input while minimizing the outputs due to the disturbance (cross-coupling 
effects) inputs. 

In each of the nine structures of Figure 22.4 it is necessary that the control ratio t; must be a member 
of the acceptable t,jet;;. All the g; and f;; must be chosen to ensure that this condition is satisfied, thus 
constituting nine MISO design problems. If all of these MISO problems are solved, there exists a fixed 
point; then yj on the left-hand side of Equation 22.40 may be replaced by #;, and all the elements of T on 
the right-hand side by t;;. This means that there exist nine tj and t;, each in its acceptable set, which is 
a solution to Figure 22.18. If each element is 1:1, then this solution must be unique. A more formal and 
detailed treatment is given in [7]. 


22.4.3 Tracking and Cross-Coupling Specifications 


The presentation for the remaining portion of this chapter is based upon not only a diagonal G matrix but 
also a diagonal F matrix. Thus, in Figure 22.4 the tj; terms, for i 4 j, represent disturbance responses due 
to the cross-coupling effect whereas the t; terms, for i = j (see Equation 22.45) is composed of a desired 
tracking term ¢, and of an unwanted cross-coupling term t,. Therefore, the desired tracking specifications 
for the diagonal MISO systems of Figure 22.4 contain an upper-bound and a lower bound as shown in 
Figure 22.6. The disturbance specification for all MISO loops is given by only an upper bound. These 
performance specifications are shown in Figure 22.19 for a 2 x 2 (in the boxed area) and for a 3 x 3 
MIMO feedback control system. 


22.4.3.1 Tracking Specifications 


Based upon the analysis of Equations 22.45 through 22.47, the specifications for the tjj responses shown 
in Figure 22.19 need to be modified as shown in Figure 22.20. As shown in this figure, a portion of 5r( jj) 
(see Figure 22.6) has been allocated [2,7] for the disturbance (cross-coupling effect) specification. Thus, 
based upon this modification and given an uncertain plant P = {P;}(j = 1,2,...,J) and the BW ow, above 
which output sensitivity is ignored, it is desired to synthesize G and F such that for all PeP 
aj, < |ti(jo)| < bi, for w < wp (22.48) 
A finite w), is recommended because, in strictly proper systems, feedback is not effective in the high- 
frequency range. 
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FIGURE 22.19 Tracking and cross-coupling specifications for a 2 x 2 (in boxed area) and for a3 x 3 MIMO system. 


22.4.3.2 Disturbance Specification (Cross-Coupling Effect) 


Based upon the previous discussion the disturbance specification, an upper bound, is expressed as 


Thus, the synthesis of G must satisfy both Equations 22.48 and 22.49. 


tcl < Lb 


(22.49) 


22.4.4 Determination of Tracking, Cross-Coupling, and Optimal Bounds 


The remaining portion of the MIMO QFT approach is confined to a 2 x 2 system. The reader can refer 
to the references for higher-order systems (m > 2). From Equation 22.39 the following equations are 


obtained: 
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FIGURE 22.20 Allocation for tracking and cross-coupling specifications for the tj; responses. 
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where 
21 
q=-—, L=quigi 
qi2 
12411 
to= 22.50 
R=TrTh ( ) 
where 
t2 
c2=-—, firz=0 
qi2 
= €214922 (22.51) 
1+L, 
where 
ty 
a=-—, lr=qng, fra =, 
921 
an Lofra + €22422 (22.52) 
2=— Ta, +b . 
where 
t 
a= (22.53) 
21 


Equations 22.50 and 22.51 correspond to the MISO systems for the first row of loops in Figure 22.4, 
and Equations 22.52 and 22.53 correspond to the MISO loops for the second row. 


22.4.4.1 Tracking Bounds 


The tracking bounds for the ii MISO system is determined in the same manner as for the MISO system of 
PART II (see Section 22.2.9.1). By use of the templates for the ii loop plant, the value of 5r(jw;)’, shown 
in Figure 22.20, is used to satisfy the constraint of Equation 22.48. 


22.4.4.2 Cross-Coupling Bounds 


From Equations 22.51 and 22.53, considering only the first row of MISO loops in Figure 22.4, the following 
cross-coupling transfer functions are obtained (see Figures 22.20 and 22.19, respectively): 


C1411 


| au| = sy fae: = Tey °° (22.54) 
€12411 = 
Itero| = 1+ <bjy=by2 --: (22.55) 


Substituting for c); and c)z into Equations 22.54 and 22.55, respectively, and replacing tz; and t22 by 


their respective upper bound values b2; and 


bo2, and rearranging these equations yields: 


1 412} Ter, 

ress ~ | qir| bar ae ( 
1 b 
| eee aoe. (22.57) 
1+1, qu | b22 
Substituting into these equations L; = 1/I; yields: 

h h 
tee 22.58 
1+],|— _ 1+h]|7— fe ( ) 


By analyzing these equations for each of the J plants over the desired BW, the maximum value My, 


that each of these equations can have, for each value of wj;(or vj), is readily determined by use of a 
CAD package. Thus, since L; = 1/l,, the reciprocals of these values yield the value of the corresponding 
M-contours or cross-coupling bounds, for w = w;(or v;), on the NC. 
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22.4.4.3 Optimal Bounds 


The points on the optimal bound for a given value of frequency and for a given row of MISO loops of 
Figure 22.4 are determined by selecting the largest dB value, for a given NC phase angle, from all the 
tracking and cross-coupling bounds for these loops at this frequency. The MIMO QFT CAD package is 
designed to perform this determination of the optimal bounds. 


22.4.5 QFT Methods of Designing MIMO Systems 


There are two methods of achieving a QFT MIMO design. Method 1 involves synthesizing the loop 
transmission function L; and the prefilter f;; independent of the previous synthesized loop transmission 
functions and prefilters. Method 2 substitutes the synthesized g; and fj; of the first (or prior) MISO loop(s) 
that is (are) designed into the equations that describe the remaining loops to be designed. For Method 2, 
it is necessary to make the decision as to the order that the L; functions are to be synthesized. Generally, 
the loops are chosen on the basis of the phase margin frequency w4 requirements. That is, the loop having 
the smallest value of w4 is chosen as the first loop to be designed; the loop having the next smallest value 
of wy is selected as the second loop to be designed; etc. This is an important requirement for Method 2. 


22.4.5.1 Method 1 


This method involves overdesign (worst-case scenario), i.e., in getting the M,, values of Equations 22.56 
and 22.57, for the 2 x 2 case, the maximum magnitude that q12 and the minimum magnitude that q1, can 
have, for each value of wj;, over the entire J LTI plants are utilized. This method requires that the diagonal 
dominance condition [2,7] be satisfied. When this condition is not satisfied, then Method 2 needs to be 
utilized. 


22.4.5.2 Method 2 


Once the order in which the loops are to be designed is designated accordingly (loop 1, loop 2, etc.), 
then the compensator g; and the prefilter fj; are synthsized. These are now known LTI functions, which 
are utilized to define the loop 2 effective plant transfer function. That is, substitute Equation 22.51 into 
Equation 22.53, then rearrange the result to obtain a new expression for ty2 in terms of g; and fi; as 


follows: 
L 1+L 
fees E 1922, /421( | (22.59) 
1+ g2422, 
where the effective loop 2 transfer function is 
1+L 
422. = god tli) where y12 = 911922 (22.60) 
(1+1L1)— yi2 912921 
Repeating a similar procedure, the expression for ty is 
_ f2282922, (22.61) 


aaa + £2922, 
Remember that a diagonal prefilter matrix has been specified. Note that Equations 22.59 through 22.61 
involve the known fi; and g;, which reduces the overdesign of loop 2. 
22.4.6 Synthesizing the Loop Transmission and Prefilter Functions 


Once the optimal bound has been determined for each L; loop, then the synthesis procedures for deter- 
mining the loop transmission and prefilter functions are the same as for the MISO analog and discrete 
systems as discussed in Sections 22.2 and 22.3, respectively. 
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22.4.7 Overview of the MIMO QFT CAD Package 


The MIMO QFT CAD package [15], implemented using Mathematica, is capable of carrying an analog 
or a discrete MISO or MIMO QFT design from problem setup through the design process to a frequency 
domain analysis of the compensated system. For analog control problems, the design process is performed 
in the s-plane. The design process is performed in either the w-plane or the s-plane using the pseudo- 
continuous-time (PCT) representation of the sampled-data system. A flowchart of the CAD package is 
given in Appendix B. 


22.5 QFT Application 


A MIMO OFT example, [18] a 2 x 2 analog flight control system (see the boxed-in loops of Figure 22.4), 
is presented to illustrate the power of this design technique. Also, this example illustrates the increased 
accuracy and efficiency achieved by the MIMO QFT CAD package [15] and the straightforward method 
for designing an analog MIMO control system. The specifications require a robust analog design for an 
aircraft that provides stability and meets time domain performance requirements for the specified four 
flight conditions (Table 22.1) and the six aircraft failure modes (Table 22.2). Table 22.3 lists the resulting 
set of 24 plant cases that incorporate these flight conditions and failure modes. For stability, a 45° phase 


TABLE 22.1 Flight Conditions 


Aircraft Parameters 


Flight Condition Mach Altitude 
1 0.2 30 
2 0.6 30,000 
3 0.9 20,000 
4 1.6 30,000 


TABLE 22.2 Failure Modes 


Failure Mode Failure Condition 

1 Healthy aircraft 

2 One horizontal tail fails 

3 One flaperon fails 

4 One horizontal tail and one flaperon fail, same side 

5 One horizontal tail and one flaperon fail, opposite side 
6 Both flaperons fail 


TABLE 22.3 Plant Models 


Flight Condition 

Failure Mode 1 2 3 4 

1 #1 #7 #13 #19 
2 #2 #8 #14 #20 
3 #3 #9 #15 #21 
4 #4 #10 #16 #22 
5 #5 #11 #17 #23 
6 #6 #12 #18 #24 
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s-domain open loop transmission for channel 2 
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FIGURE 22.21 Open-loop transmissions on NC. 


margin is required for each of the two feedback loops. Frequency domain performance specifications, 
when met, result in the desired closed-loop system performance in the time domain. The frequency 
domain specifications are shown as dashed lines on the Bode plots of Figure 22.22. 

The specifications, the plant models [18] for the 24 cases, and the weighting matrix are entered into the 
QFT CAD package. The automated features accessed through the designer interface of the CAD package 
result in synthesizing the compensators gj (s) and go(s) in the manner described in Section 22.2.10. That is, 
the nominal loop transmission functions L1)(s) = gi(s)qiig(s) and La, (s) = g2(s)qz29(s) are synthesized 
(or shaped) so that they satisfy their respective stability bounds and their respective optimal bounds 
By, (jw;) and Bp, (jw;). Note that qi1, and q22, are the nominal plant transfer functions. The first step in 
a validation check is to plot the loop transmission functions Lz, (s), where t = 1,..., 24, for all 24 cases on 
the NC. This is accomplished by a CAD routine, as shown in Figure 22.21 for the purpose of a stability 
check (m.p. plants). As is seen, none of the cases violate the M, stability contour (the dark ellipse). In this 
design, when synthesizing L2, (s) a trade-off exists between performance and bandwidth. In this example, 
the designer chooses to accept the consequences of violating the disturbance bound for w = 2 rad/s. With 
L,(s) and L2,(s) synthesized, the automated features of the CAD package expedite the design of the 
prefilters f11(s) and fy2(s). 

For the second step in the design validation process, the 2 x 2 array of Bode plots shown in Figure 22.22 
is generated, showing on each plot the 24 possible closed-loop transmissions from an input to an output 
of the completed system. The consequence of violating the channel 2 disturbance bound for w = 2 rad/s 
is seen where the closed-loop transmissions violate bz), denoted by dashed line, beginning at w = 2 rad/s. 
Violation of performance bounds during loop-shaping may result in violation of the performance speci- 
fications for the closed-loop system. 

As seen in Figure 22.22, a robust design has been achieved for this 2 x 2 MIMO analog flight control 
system. The time domain results, although not drawn, meet all specifications. 
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FIGURE 22.22 Closed-loop transmissions for an analog design system. 
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22.6 Appendix A 


CAD flowchart for MISO QFT design. 
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22.7 Appendix B 


MIMO QFT flowchart for analog and discrete control systems [15]. 
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23.1 Introduction 


The so-called servomechanism problem is one of the most basic problems to occur in the field of automatic 
control, and it arises in almost all application problems of the aerospace and process industries. In the ser- 
vomechanism problem, it is desired to design a controller for a plant (or “system”) so that the outputs of the 
plant are independent, as much as possible, from disturbances which may affect the system (i.e., regulation 
occurs), and also such that the outputs asymptotically track any specified reference input signals applied to 
the system (i.e., tracking occurs), subject to the requirements of maintaining closed-loop system stability. 

This chapter examines some aspects of controller synthesis for the multivariable servomechanism 
problem when the plant to be controlled is subject to uncertainty. In this case, a controller is to be designed 
so that desired regulation and tracking takes place in spite of the fact that the plant dynamics or/and 
parameters may vary by arbitrary, large amounts, subject only to the condition that the resultant closed- 
loop perturbed system remains stable. This problem is called the robust servomechanism problem (RSP). 


23.2 Preliminary Results 
23.2.1 Plant Model 


The plant to be controlled is assumed to be described by the following linear time-invariant (LTI) model: 


x= Ax+Bu+ Ew 
23-1 
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y=Cx+ Dut Fo 
Ym = CnxX + Dnu+ Fno (23.1) 
€=V— Vref 
where x € R” is the state, u € IR” are the inputs that can be manipulated, y € R’ are the outputs that are 
to be regulated and y», € R™ are the outputs which can be measured. Here w € R® correspond to the 
disturbances in the system, which in general cannot necessarily be measured, and e € R’ is the error in 


the system, which is the difference between the output y and the reference input signal y,e¢, in which it is 
desired that the outputs y should track. 


23.2.2 Class of Tracking/Disturbance Signals 
It is assumed that the disturbances w arise from the following class of systems: 
M=Ain, ©=Cim;s 11¢€R" (23.2) 
and that the reference input signals y;er arise from the following class of systems: 
M2 = Aan, p=Com2, Yret = GP; m2 €R” (23.3) 


It is assumed for nontriviality that sp(A1) C Ct, sp(A2) C Ct, where sp(-) denotes the eigenvalues of 
(-) and Ct denotes the closed right complex half-plane. It is also assumed with no loss of generality that 
(Cy, Az), (C2, Az) are observable and that rank (F) = rankC, = dim(w), rank G = rank C2 = dim(p). 

This class of signals is quite broad and includes most classes of signals that occur in application 
problems, e.g., constant, polynomial, sinusoidal, polynomial-sinusoidal, etc. 

The following definitions will be used in the development to follow. 


Definition 23.1: 


Given the systems represented by Equations 23.2, and 23.3, let {h1,2,..-,p} be the zeros of the least 
common multiple of the nominal polynomial of A, and minimal polynomial of Az (multiplicities repeated), 
and call 

A:= Orr, das.- +s Ap} (23.4) 


the disturbances/tracking poles of Equations 23.2 and 23.3. 


Definition 23.2: 


Given the model represented by Equation 23.1, consider the system 


k= Ax+Bu; ueR", yEeR’, xe R" 


y=Cx+Du 
Then d € C is said to be a transmission zero (TZ) [3] of (C, A, B, D) if 


A-dI B 


rank C D 


<n+min(r,m) 


In particular, the transmission zeros are the zeros (multiplicities included) of the greatest common 
A—dI 4 


divisor of all [n + min(r, m)] x [n+ min(r, m)] minors of | C pI: 
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Definition 23.3: 


Given the system (C, A, B, D), assume that one or more of the transmission zeros of (C, A, B, D) are contained 
in the closed right complex half-plane; then (C,A, B,D) is said to be a nonminimum-phase system. If 
(C, A, B, D) is not nonminimum phase, then it is said to be a minimum-phase system. 


23.2.3 Robust Servomechanism Problem 


The robust servomechanism problem (RSP) for Equation 23.1 consists in finding an LTI controller that 
has inputs yn, Vref and outputs u for the plant so that: 


1. The resultant closed-loop system is asymptotically stable. 
2. Asymptotic tracking occurs; that is, 


Jim e(t)=0, Vx(0)ER", Vni(0)ER™, Wn2(0) eR” 

00 
for all controller initial conditions. 

3. Condition 2 holds for any arbitrary perturbations in the plant model Equation 23.1 (e.g., plant 
parameters or plant dynamics, including changes in model order) that do not cause the resultant 
closed-loop system to become unstable. 


In this problem statement, there is no requirement made regarding the transient behavior of the 
closed-loop system; thus, the following problem statement is now made. 


Perfect RSP 


Given the plant represented by Equation 23.1, it is desired to find a controller such that: 


1. It solves the RSP for the class of disturbances/tracking signals given by Equations 23.2 and 23.3. 
2. The controller gives perfect error regulation when applied to the nominal plant model Equa- 
tion 23.1, ie., given x(0), z1(0), Z2(0), located on the unit sphere, with n(0) = 0, where 1(0) is the 
initial condition of the servo-compensator (see Equation 23.9), then Ve > 0, there exists a controller 
(parameterized by ¢) that satisfies property 1 and has the property that fe e’(t)e(t) dt <«, with 
no unbounded peaking occurring in the response of e; i.e., there exists a constant p, independent 
of €, such that sup |e(t)| < p. 
t>0 
Thus, in the perfect RSP, arbitrarily good transient error, with no unbounded peaking in the error response 
of the system, can be obtained for any initial condition of the plant and for any disturbance/tracking signals 
that belong to Equations 23.2 and 23.3. 


23.3 Main Results 


The following results are obtained concerning the existence of a solution to the RSP [5,6]. 


Theorem 23.1: 


There exists a solution to the RSP for Equation 23.1, if and only if the following conditions are all satisfied: 


1. (Cy, A, B) is stabilizable and detectable. 
2. m>r. 
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3. The transmission zeros of (C, A, B, D) exclude the disturbance/tracking poles j,i = 1,2,...,p. 
4. y C ym ie., the outputs y are measurable. 


Remark 23.1 


The conditions 2 and 3 are equivalent to the condition: 


A-hl B 


rank | C D 


]=*+" i=1,2,...,p (23.5) 


The following existence results are obtained concerning the existence of a solution to the perfect 
RSP [12]: 


Theorem 23.2: 


There exists a solution to the perfect RSP for Equation 23.1, if and only if the following conditions are all 
satisfied: 


1. (Cy, A) is detectable 

2. m>r 

3. (C,A, B, D) is minimum phase 
4. yC¥m 


Remark 23.2 


If m = 1, the above conditions simplify to just conditions 3 and 4. 
The following definitions of a stabilizing compensator and servo-compensator are required in the 
development to follow. 


Definition 23.4: 


Given the stabilizable, detectable system (Cy, A,B,D) obtained from Equation 23.1, an LTI stabilizing 
compensator 


&= AE+ Arym 


(23.6) 
u=Ki§+ Koym 


is defined to be a controller that asymptotically stabilizes the resultant closed-loop system, such that “desired” 
transient behavior occurs. 


This compensator is not a unique device and may be designed by using a number of different techniques. 


Definition 23.5: 


Given the disturbance/reference input poles )j, i= 1,2,...,p, the matrix C € R?*? and vector y € RP are 
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defined by 
0 1 0 0 : 
0 0 1 aa 0 
C:= , . ee ae : (23.7) 
: ; : : ‘ 
—8; —8 —83 —8p i 


P 
where the coefficients 8;, i= 1,2,...,p are given by the coefficients of the polynomial []o — hi); ie, 


i=1 
P 
We 8p lg. +82448) =] [Q-2i) (23.8) 
i=1 


The following compensator, called a servo-compensator, is of fundamental importance in the design of 
controllers to solve the RSP [5]. 


Definition 23.6: 


Consider the class of disturbance/tracking signals given by Equations 23.2 and 23.3, and consider the system 
of Equation 23.1; then a servo-compensator for Equation 23.1 is a controller with input e € R' and output 
1 € R? given by 


i=Cy+B*e (23.9) 
where 
C* := block diag (C,C,...,C) (23.10) 
——$—"” 
B* := block diag (y, y,...,Y) (23.11) 
— 


r 
where C, y are given by Equation 23.7. 
The servo-compensator is unique within the class of coordinate transformations and nonsingular input 


transformations. 
Given the servo-compensator of Equation 23.9, now let DeR"™’? be defined by 


D := block diag (8,8, ...,8) (23.12) 
——$ 
where § € R!*? is given by: 
§:=(100...0) (23.13) 


The servo-compensator has the following properties: 


Lemma 23.1: [12] 


Given the plant represented by Equation 23.1, assume that the existence conditions of Theorem 23.1 all 
hold; then 
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U( )-(e% &)-(s0)] 


is stabilizable and detectable and has centralized fixed modes [8] (i.e., those modes of the system 
that are not both simultaneously controllable and observable) equal to the centralized fixed modes of 


(Cig A BeBe), 
A 0) (B 
[omm(ec &):(en)| 


2. The transmission zeros of 
are equal to the transmission zeros of (C, A, B, D). 


1. The system 


23.3.1 Robust Servomechanism Controller 


Consider the system of Equation 23.1, and assume that the existence conditions of Theorem 23.1 hold; 
then any LTI controller that solves the RSP for Equation 23.1 consists of the following structure [6] (see 
Figure 23.1): 


u=t+Ky (23.14) 


where 1 € R’? is the output of the servo-compensator (Equation 23.9), and & is the output of a stabilizing 
compensator S with inputs ym, ref, 1, Us Where S, K are found to stabilize and give “desired behavior” to 
the following stabilizable and detectable system: 


( i ) : E o (;) . ied) o 


Cn 0 Dy (23.15) 
od x 
Yn = 0 I ( ) + 0 Vref 
o o| " I 


where, from Lemma 23.1, the centralized fixed modes (if any) of 


Cn 0 A 0 B 
0 I)’|B*C C*l]’\BtD 
are equal to the centralized fixed modes of (C,, A, B); i-e., there always exists a coordinate transformation 


and nonsingular input transformation, by which any controller that solves the RSP for Equation 23.1 can 
be described by Equation 23.14. It is to be noted that this controller always has order > rp. 


Servo-compensator | 


Vn 
Vref + e a + Uu 
‘o >) >| n=C*n+Bre > Ki >& > Plant >| T >y 


Stabilizing compensator /X— 


FIGURE 23.1 General controller to solve robust servomechanism problem. 
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Properties of Robust Servomechanism Controller 


Some properties of the robust servomechanism controller (RSC) represented by Equation 23.14 are as 
follows [5]: 


1. In the above controller, it is required to know only the disturbance/reference input poles 
(01, d2..--5Ap}s Le., it is not necessary to know E, F of Equation 23.1 nor Aj, Az, C1, C2, G 
of Equations 23.2 and 23.3. 

2. A controller exists generically [2] for almost all plants described by Equation 23.1, provided that 
(a) m> r, and (b) the outputs y can be measured; if either condition (a) or (b) fails to hold, there 
is no solution to the RSP. 


23.3.2 Various Classes of Stabilizing Compensators 


Various special classes of stabilizing compensators S that can be used in the RSC (Equation 23.14) are as 
follows. It is assumed that the existence conditions of Theorem 23.1 are all satisfied in order to implement 
these proposed controllers. 


Multivariable Three-Term Controller 
(See Figure 23.2). In order to use this controller, it is assumed that: 
1. The plant of Equation 23.1 is open loop asymptotically stable. 


2. The disturbance/tracking poles {)j,X2,...,p} are of the polynomial-sinusoidal type; i.e., it is 
assumed that Re(A;) = 0,i=1,2,...,p. 


If these assumptions hold, then the following generalized three-term controller solves the RSP [9]: 


u= (Ko + Kys)e+ Koy 


(23.16) 
h=C*n+ Bre 


An algorithm is given [7], that shows that a stabilizing Kz, with Kp = 0, K; = 0, can always be found for 
this controller. 


23.3.3 Complementary Controller 


(See Figure 23.3) In order to use this controller, it is assumed that the plant of Equation 23.1 is open loop 
asymptotically stable. If this assumption holds, then the following controller, called a complementary 


y, aa e Ym 
f + ig 
‘0 > 7=C'n+Bre LL K, DP Plant > T >y 


Ly} K+Kjs g 


FIGURE 23.2 Generalized three term controller to solve the robust servomechanism problem. 
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i: 
s al ye 
Ko “&= Ax + Bu 


FIGURE 23.3 Complementary controller to solve the robust servomechanism problem. 


controller, will solve the RSP [5]: 


u= Kox + Kin 
1 =C*n+B*e (23.17) 
R= AR+ Bu 


where (Ko, Kj) are found to stabilize the stabilizable system 


in A 0] - B 
ale o|e+( B*D jo (23.18) 


using state feedback; ie., u = (Ko K1)x. 


23.3.4 Observer-Based Stabilizing Controller 


(See Figure 23.4) No additional assumptions are required in order to implement this controller. The 
controller is given as follows [9]: 
u=Kox+Kin 
H=C'n+Bre (23.19) 
& = {A+ (B—AD)Ko — ACm}% + (B— ADm)Kin + AYm 
where (Ko, K1) are found to stabilize the system of Equation 23.18, and A is an observer gain matrix found 


to stabilize the system matrix (A — AC,,) where (Cy, A) is detectable. (A reduced-order observer could 
also be used to replace the full-order observer in Equation 23.19.) 


Vref + e 1 + Uu 
‘O >) > 1=Cn+ Bre K, P& > Plant > T >y 
g 
Ko Observer for 
x & = Ak + Bu 


FIGURE 23.4 Observer-based stabilizing compensator to solve the robust servomechanism problem. 
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23.4 Applications and Example Calculations 


We initially demonstrate the theory by considering the design of a controller for a nonminimum-phase 
plant (Example 23.1), and a minimum-phase plant (Example 23.2), and thence conclude with a case study 
on the control of a nontrivial distillation column system. 


Example 23.1: Nonminimum-Phase System 


Consider the following system 
x= é ‘) x+ (i) u 
0 Oo 1 (23.20) 


which is open loop unstable and nonminimum phase (with a transmission zero at 1), and in which it 
is desired to design a controller to solve the RSP for the case of constant disturbances and constant 
reference input signals. In this case, the disturbance/tracking poles = {0}, and it can be directly 
verified that the existence conditions for a solution to the problem are satisfied from Theorem 23.1. 
In the controller design, it is initially assumed that the control input should not be “excessively large.” 


Controller Development 


On applying the servo-compensator of Equation 23.9 for constant disturbances/reference input signals 
to Equation 23.20, the following system is obtained: 


: 2 -1 0 1 0 
GE 0 0 0 (*)+ L}o+] 0 | yer 
3 1 0 of \Y \o ai 
x 
y=(1 0 0) (*) +1 (23.21) 
and, on minimizing the performance index [12] for Equation 23.21 
[o,@) 
Je = (e+ei'u)dt, €=1 (23.22) 
0 
the following controller is obtained: 
t 
“= (ky ky)x + kf (y — Vref) (23.23) 
0 


where 


ky =—-13.77, ky, =8.721, k=1.000 


and where x is the output of an observer for Equation 23.20. On using a reduced-order observer with 
observer pole = —1, the following controller is thence obtained. 
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u = —22,498y + yn + 8.72140 
iy =Y— ref (23.24) 
6 = 16.4430 + 2n — 41.996y 

which, when applied to Equation 23.20, gives the following closed-loop poles: 


—0.552 + j0.456 
—1.00 
—1.95 


Properties of Controller 


On applying the controller of Equation 23.24 to the plant of Equation 23.20, the following closed-loop 
system is obtained: 


S098 B72 14 0 ~22.49 
: 22.498 0 1. 8.7214| (~ 0 ~22.49 
A oe 0 0 0 MYT] _y preety 
ie 41.996 0 2 16.443] \° 0 —41.99 
x 
y=[1 0 0 Oj n+ 
oO 


x 
u=[—22.498 0 1. 8.7214] | n | + (—22.498)o 
(o) 


which gives the Bode magnitude response between e, u and input yrer in Figure 23.5, e, u and input w 
in Figure 23.6, and unit-step responses in y;e¢ in Figure 23.7 and in w in Figure 23.8, with zero initial 
conditions. It is seen that satisfactory tracking/regulation occurs using the controller. 


Response of Closed-Loop System to Unbounded Reference Input/Disturbance Signals 


According to Theorem 3 in [12], the RSC of Equation 23.14 has the property of not only achieving 
asymptotic tracking/regulation for the class of constant reference input/disturbance signals, but also 
bringing about exact asymptotic tracking/regulation for unbounded signals that have the property that: 


lim Veet (t) =0 

a, (23.25) 
lim w(t) =0 
too 


To illustrate this result, Figure 23.9 gives the response of the closed-loop system for the case when the 
tracking input signal is given by 
yrep(t)=t!/4, t>0 (23.26) 
with zero initial conditions. It is seen that exact asymptotic tracking indeed does take place using this 
controller. 


Optimum Response of Nonminimum-Phase System 


The RSC of Equation 23.24 was obtained so that the magnitude of the control input signal is constrained 
by letting « = 1 in the performance index of Equation 23.22. It is of interest to determine what type of 
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(a) Bode magnitude of closed loop system from y,.¢ to error 
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FIGURE 23.5 Bode plot magnitude of closed-loop system using robust servomechanism controller from reference 
input signal: (a) to error; (b) to control input. 
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FIGURE 23.6 Bode plot magnitude of closed-loop system using robust servomechanism controller from disturbance 
signal: (a) to error; (b) to control input. 
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FIGURE 23.7 Response of closed-loop system using robust servomechanism controller for unit step in reference 
input signal: (a) output; (b) input. 
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FIGURE 23.8 Response of closed-loop system using robust servomechanism controller for unit step in disturbance 
input signal: (a) output; (b) input. 
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(a) Output response of closed loop system to unbounded y,¢¢ 
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FIGURE 23.9 Response of closed-loop system using robust servomechanism controller for unbounded reference 
input signal = ¢!/4 : (a) output; (b) input. 


transient performance can be obtained for the system if the controller is designed without any regard 
for the magnitude of the control input signal; i.e., by letting « = 10~8, say, in the performance index 
(Equation 23.22). According to Theorem 23.2, “perfect control” cannot be achieved for the system; in 
particular, if the plant’s and controller’s initial conditions are equal to zero and w = 0, then the limiting 
performance that can be achieved is given by [13] 


i 
. 3 1, 
lim Je = 2 jer ref (23.27) 


i=l 71 


where {7, i= 1,2,...,]} are the nonminimum TZ of the system, and J. is given by Equation 23.22. In 
this case, / = 1, \} = 1, so that the limiting performance index is given by lim J, = See 
«>0 


The following optimal controller is now obtained on putting « = 10~® in the performance index of 
Equation 23.22: 


u = (—2.043 x 10* 2.029 x 104)x + 104 


; (23.28) 
N=) — Vref 
This controller results in the following closed-loop system poles: 
—70.7 +j70.7 
—1.00 
with the following optimal performance index, for x(0) = 0, w = 0: 
J =2.014y2,, (23.29) 


which is “close” to the optimal limiting performance index of Dye gs 
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Output response of closed loop system for step input in y,o 


-l 1 1 1 L 1 1 1 1 i 
0 0.5 1 15 2 2.5 3 3.5 4 45 5 
Time (s) 
(b) Input response of closed loop system for step input in y,¢¢ 
20 T T T T T 
0 Ln 
x -20 4 
~40 4 
60 L 1 1 1 1 
0 0.5 1 15 2 2.5 3 
Time (s) 


FIGURE 23.10 Response of closed-loop system using robust servomechanism controller with high gain for unit 
step in reference input signal: (a) output; (b) input. 


A response of the closed-loop system using the controller of Equation 23.28 for a unit step in y,e¢, with 
zero initial conditions for the plant, is given in Figure 23.10. It is seen that the response of the closed-loop 
system is only about twice as fast as that obtained using the controller of Equation 23.24, in spite of the 
fact that the control signal is now some 40 times larger than that obtained using Equation 23.24, which 
confirms the fact that nonminimum-phase systems are fundamentally “difficult to control.” 


High-Gain Servomechanism Control 
The same example as considered in the previous sections will now be considered, except that a high- 


gain servomechanism controller (HGSC) [12], which is simpler than an RSC, will now be applied and 
compared with the RSC of Equation 23.24. 


Plant Model 
<6 aC) 
~ \o 0 1 (23.30) 


Controller Development 


Given the cheap control performance index 


i= | (yy +eu'u) dt (23.31) 
0 
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let € = 1; in this case, the optimal control that minimizes J, for the system of Equation 23.30 with w = 0 
is given by 
u=kx, k=(—10.292 5.6458) (23.32) 
which results in the closed-loop system of Equations 23.30 and 23.33 having poles given by (—0.457, 
=9,19), 
On letting K := —[c(A + Bk)~!B]~! = —1, the HGSC is now given by [12]: 


U=K (rep —y) + (kK + Ko)k (23.33) 


where x is the output of an observer for Equation 23.30. On choosing a reduced-order observer with 
observer gain = —1, and simplifying, the following controller is finally obtained: 


High-Gain Servomechanism Controller 


U = —Yrep — 15.94y — 5.6468 


: (23.34) 
£ = 10.295 + 28.87y + 2yrer 


Properties of HGSC 
On applying Equation 23.34 to Equation 23.30, the following closed-loop system is obtained: 
—13.94 —1l —5.646 —1 0 


(:)- S95 94.° 0 = 5646 ({)+ = bee Oo. fe 
28.87 0 10.29 2 28.87 


e=[1 0 0] @ = lye PL 


yep o (2) +10 
u=[-15.94 0 —5.646] (£) - 17 - 15940 


which has closed-loop poles given by (—0.457, —1.00, —2.19). The Bode magnitude response of this 
system with respect to output e, u and input yye¢ is given in Figure 23.11. The unit-step function response 
of this system for an increase in y;er with zero initial conditions is given in Figure 23.12. It is seen that 
satisfactory tracking occurs using this controller. 


Comparison of Robustness Properties of HGSC to RSP 


Although the HGSC of Equation 23.34 is “simpler” than the RSC of Equation 23.24, the outstanding 
advantage of the RSP is that it is robust; ie., it provides exact asymptotic tracking/regulation for any 
perturbations of the plant model, that do not destabilize the perturbed closed-loop system. As an example 
of this behavior, consider the HGSC (Equation 23.34) and RSC (Equation 23.24) controlling the following 
(slightly) perturbed model of Equation 23.20: 


=( 2i)+()} 
~\o -0.1 1 (23.35) 


y=(1 0)x+o 


Perturbed Model 


In this case, the resultant perturbed closed-loop system remains stable for each of the controllers, but the 
HGSC no longer provides tracking; e.g., Jim y(t) = 0.3 when y;ef = 1. In contrast, it may be verified that 
00 


the RSC (Equation 23.24) still provides exact tracking when applied to Equation 23.35. 


Bode magnitude of closed loop system from y,.¢ to error 
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FIGURE 23.11 Bode plot magnitude of closed-loop system using high-gain servomechanism controller from refer- 


ence input signal: (a) to error; (b) to control input. 
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Output response of closed loop system for step input in y,o¢ 
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FIGURE 23.12 Response of closed-loop system using high-gain servomechanism controller for unit step in reference 


input signal: (a) output; (b) input. 
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Example 23.2: Minimum-Phase System 
Consider the following system, which is a model of a pressurized head box in paper manufacturing: 


0.395 0.01145 0.03362 1.038 
—0.011 0 0.000966 0 


1 0 
y=(4 1) Ym =Y (23.36) 


which has open-loop eigenvalues = (—3.19 x 1074, —3.95 x 1071). In this case, the plant represented 
by Equation 23.36 has no transmission zeros, which implies that it is minimum phase, and so there 
exists a solution to the RSP for the class of constant disturbances/reference input signals such that 
perfect control occurs (see Theorem 23.2). 


The following development illustrates how “perfect control” ina minimum-phase system results in arbi- 
trarily good approximate error regulation (AGAER) [12] occurring; i-e., although the servo-compensator 
is designed to give exact asymptotic error regulation for constant disturbances/reference input signals, 
AGAER will occur for other classes of disturbances/reference input signals, e.g., for the class of sinusoidal 
signals yrer = y sin(ot). 

The following perfect controller is now obtained: 


t 
us Kay +K | (Y= Vret) At (23.37) 
0 


which, on minimizing the performance index 


CO 
n= [ (ee + ew!) dt (23.38) 
0 
gives 
—4,33 —4549 —3719 —104 
Ky = + he fore = 107° 
® Gee ) ee a ue 
—13.8 1.439 x 104 —338.8 —9.994 x 104 
R= ee fore=10'° (23.3 
: ‘ee 477.3 ) Roe 3388 ) oe ee?) 


It may now be demonstrated that AGAER occurs; e.g., when the reference input signal y,.¢ = y sin(wt), 
w = 0.1 is applied, we obtain the following error coefficient [12]: 


1.4x107-3 10x 1074 =! 
K°(jo) = fore =10°° 
0K; Go) ( 10x 10-4 46x 1072 ) ae 


44x10-4 1.0x 1075 
=(~ fore = 107!° 23.40 
CF x 10-5 14x ae) mae a) 


which implies that excellent (but approximate) tracking occurs as € — 0 for the tracking signal y,er = 
ysin(wt), o = 0.1. 


Stability Robustness Concerns 


The previous example studies have ignored “stability robust concerns”; i.e., the recognition that any 
plant model is really only an approximation to a model of the actual physical system, and hence that a 
given physical system may become unstable under feedback if the model used is sufficiently inaccurate. 
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The following shows that for some classes of systems, a system may be relatively insensitive to plant 
perturbations, whereas for other classes of systems, a system may be highly sensitive to plant perturbations. 
This implies that any controller that is designed to solve the RSP must always take into account stability 
robustness considerations for the particular problem being studied. 

Consider the head box example (Equation 23.36) and the controller 


t 
u=Koy+K [ (Y= Yret) At (23.41) 
0 


which has been designed for the class of constant disturbances/reference input signals. It is now desired 
to determine the optimal controller gain matrices Ko, K that minimize the performance index 


CO 
iP 2) (’e+enindt ¢e=10° (23.42) 
0 


with and without a gain margin (GM) constraint [10] imposed on the system. This constraint can be 
imposed by using the computer-aided design (CAD) approach of Davison and Ferguson [9]. Let denote 
the optimal cost matrix of Equation 23.44, and define Jop; = trace (I’). The following results are now 
obtained (see Table 23.1) 

In this case, the optimal controller that minimizes J; (Equation 23.44), subject to a fairly demanding 
GM constraint of (0.2,2) is only about “two times slower” than the optimal controller, which does not 
take gain margin into account; i.e., the system is relatively insensitive to plant perturbations. 

Consider now, however, the following system: 


(23.43) 


and the controller of Equation 23.43. It is desired now to find optimal controller parameters Ko, K so as to 
minimize the performance index J, (Equation 23.44) with and without a modest GM of (0.9,1.1) applied. 
The following results are obtained from Davison and Copeland [10] (see Table 23.2). 

In this case, it can be seen that a very modest demand that the controller should have a GM of only 
10% has produced a dramatic effect in terms of the controller. The controller obtained with the 10% GM 
constraint being imposed has a performance index that is some 10° times “worse” than the case when 
no GM constraint is imposed; i-e., the system is extremely sensitive to plant perturbations. Thus, this 
example emphasizes the need to always apply some type of stability robustness constraint when solving 
“high-performance controller” problems. 


TABLE 23.1 Optimal Controller Parameters Obtained 


GM Constraint Jopt (Ko, K) Closed-Loop Poles 
4.33 4550 371.9 9993 —2.20 + j2.20 
None 0.469 
—138 158.7 —9993 371.9 —72.1 + j72.1 
—1.43 + j1.51 
2330 245 6700 281 
(0.2,2) 0.700 —5.31 
504 820 1960 1580 


—437 
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TABLE 23.2 Optimal Controller Parameters Obtained 


GM Constraint Jopt (Ko, K) Closed-Loop Poles 


58.1 15.2 326 9995 —10.14 10.1 
None 0.100 ; 
220 —185 9995  —326 —700 + j700 
—0.0538 
—0.600 —0.962 —0.636 0.109 
(0.9,1.1) 24.4 —0.350 
0.740 0.759 0.620 —0.380 : 
—0.897 + j4.21 


More Complex Tracking/Disturbance Signal Requirements 


The following example illustrates the utilization of more complex servo-compensator construction. In 
this case, the head box problem modeled by Equation 23.36 is to be controlled for the class of distur- 
bance/reference input signals that have the structure: 


Vref = V1 Sin(w1t) + y2 sin(wat) + y3 sin(w3f) (23.44) 


where 1 = 1, 2 = 31, w3 = 50, and yj, y2, y3 are arbitrary real two-dimensional vectors. 
From Theorem 23.1, there exists a solution to the problem, and the servo-compensator is now given 
from Equation 23.9 by: 


0 

I 

: : 0 I 0 I 0 I 0 
1} = block diag (oar i) ‘ a i) : Gee Al n+ I (vy — ret) (23.45) 

0 

I 

and the following controller is now obtained: 

u=Koy+ (K, Kz K3)y (23.46) 


In this case, the following performance index 
- ad ™: 
if =I {oo 1/)Q () +eul dt (23.47) 
0 


Q:=(0 I 0101 0/0 I 0107 sO) 
«= 10° 


where 


(23.48) 


is to be minimized in order to determine the optimal controller parameters (Ko, Ki, K2, K3). As before, 
let T denote the optimal cost matrix of Equation 23.47, corresponding to a minimization of J<, and define 
Jopt = trace(I’). The following results are now obtained: 


Jopt = 36.3 


and the optimal Ko, K), Kz, K3 are given in Table 23.3. 

The eigenvalues of the resultant closed-loop system using this controller are given in Table 23.4. 

In this case, the response of the resultant closed-loop system for a triangular tracking signal of period 4 
seconds is given in Figure 23.13. It is seen that the tracking performance of the system is excellent; it is to 
be noted that the dominant harmonics of this triangular wave are given by 1, 31, 51 rad/s, and thus the 
servo-compensator is approximating the tracking of a triangular wave, in this case, by tracking/regulating 
the main harmonic terms of the periodic signal. 
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TABLE 23.3 Optimal Value of Ko, Ki, Kz, K3 Obtained 


Ko= 1.8047e+00 2.1315e+03 
—5.8028e+01 8.0400e+01 


K,=  —3.3062e+02 8.0559e+03 4.0912e+01 1.8825e+03 
9.7821e+03 2.4767e+02 6.5109e+02 8.0833e+01 
Ky = = —2.6297e+02 9.7588e+03 2.1678e+01 2.2892e+02 
8.2362e+03 3.2042e+-02 6.0072e+02 9.6246e+00 
K3= —1.8240e+02 9.909 1e+03 1.7523e+01 8.3041le+01 
5.7704e+03 3.2592e+02 5.1951le+02 3.3729e+00 


23.4.1 CAD Approach 


In designing controllers for actual physical systems, it is often important to impose on the controller 
construction constraints that are related to the specific system being considered. Such constraints can be 
incorporated directly using a CAD approach (e.g., see [9]). The following example illustrates the type of 
results that may be obtained. 

Consider the head box problem modeled by Equation 23.36, controlled by the controller 


t 
u=Koy+ x | (vy — Vref) dt (23.49) 
0 


and assume that it is desired to determine the controller gain matrices Ko, Ki, so as to minimize the 
performance index: 


CO 
k= / (le+eiin) dt, «=10% (23.50) 
0 
such that all elements of Ko, K satisfy the constraint: 
[ij] < 100 


This constraint could arise, for example, in terms of attempting to regulate the control signal magnitude 
level for a system. The following results are obtained in this case [9] (see Table 23.5). 


TABLE 23.4 Closed-Loop Eigenvalues 


—1.5016e+01 2.8300e+01i 
—1.5016e+01 —2.8300e+01i 
—3.0344e+01 

—1.2005e—01 1.3596e+01i 
—1.2005e—01 —1.3596e+01i 
—1.9565e—02 1.5708e+01i 
—1.9565e—02 —1.5708e+01i 
—5.4357e—02 9.4249e+00i 
—5.4357e—02 —9.4249e+00i 
—3.5863e—02 6.7460e+00i 
—3.5863e—02 —6.7460e+00i 
—1.0298e+00 

—4.4094e—01 3.2555e+00i 


—4.4094e—01 —3.2555e+00i 
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Response of output y, for triangular y,,-, signal 
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FIGURE 23.13 Response of closed-loop system for head box example using robust servomechanism controller for 
triangular reference input signal. 


23-22, Control System Advanced Methods 


TABLE 23.5 Results Obtained 


Constraint Jopt (Ko, K) Closed-Loop Poles 
4.33 4550 372 9993 —2.20 + j2.20 
None 0.516 
—138 159 —9993 372 —72.14572.1 
70 100 100 100 —0.157 + j0.330 
|kij| < 100 22.2 
—2.6 78 —29 22: —2.61 + j5.62 


Consider now the following model, which approximately describes the behavior of a DC motor: 


[77535 x 10? 5.163 xt)? Jug [4+] 
~ | —209.4 —198.1 188.7 0 (23.51) 
y= Obs ym=x 


and consider the following controller: 


t 
4 =Koym-+K [ (y — Vref) dt (23.52) 
0 


where (Ko, K) are to be obtained so as to minimize the performance index 
CO 
k= / (’e+i/in) dt, €=107% (23.53) 
0 


subject to the constraint that the damping factor of the closed-loop system should have the property that 
¢ > 1, in order to prevent an excessively oscillatory response, say. The following results are obtained [9] 
(see Table 23.6). 


Discrete Systems 


The previous results have considered a continuous system. For discrete-time systems, equivalent condi- 
tions for the existence of a solution to the RSP and the necessary controller structure can be obtained 
(e.g., see [4]). The following example illustrates this point. 

Consider the following discrete system: 


ve, — {0:9812 0), ( 4877 4877) | 
I= 9 0.09048) “** \-1.1895 3.369) °* 


= 1 0 XL: — 


in which it is desired to solve the RSP for the class of constant disturbances/constant tracking signals. In 
this case, the above system is controllable and observable and has no transmission zeros, so that there 


(23.54) 


TABLE 23.6 Results Obtained 


Constraint Jopt (Ko, K) Closed-Loop Poles 


5 116 +161 
None 9.93;5 [—98.1 —149 —10000] 
249 
7 153+ j153 
t>1 1.00;5  [-109 —1.62 —9480] i 
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exists a solution to the RSP; the servomechanism controller becomes in this case: 


ug = Koyg + Kn 


(23.55) 
Nk+l = Nk + Vk — Vref 
and, on minimizing the performance index 
[e,0) 
Je = Deer + (uit — Ky! nea — ua €= 10-* (23.56) 
k=1 
the following controller gain matrices are obtained: 
Fang = 6.00 
(Ko. Kar = | 7029501239, 0.152 0.219 (23.57) 
eae a said I= 17 0.239 0.0535 —0.219 


The closed-loop poles obtained by applying the above controller to the plant modeled by Equation 23.54 
are, in this case, given by —8.4 x 10-10 +j1.6 x 107°, 7.2 x 1071! +j8.1 x 107°); ive., a “dead-beat” 
closed-loop system time response is obtained. 


23.4.2 Case Study Problem—Distillation Column 


The following model of a binary distillation column with pressure variation is considered: 


x= Ax+ But Ew 
(23.58) 
y=Cx 


where (C, A, B, E) are given in Table 23.7. Here y; is the composition of the more volatile component in 
the bottom of the column, y2 is the composition of the more volatile component in the top of the column, 
and y3 is the pressure in the column; « is the input feed disturbance in the column; and u the reheater 
input, uz the condensor input, and u3 the reflux in the system. 


Eigenvalues and Transmission Zeros of Distillation Column 


The open-loop eigenvalues and transmission zeros of the distillation column are given in Table 23.8, 
which implies that the system is minimum phase. 

It is desired now to find a controller that solves the RSP problem for this system for the case of constant 
disturbances and constant reference input signals. In this case, the existence conditions of Theorem 23.1 
hold, so that a solution to the problem exists; in particular, there exists a solution to the “perfect control 
robust servomechanism” problem (see Theorem 23.2) for the system. 


Perfect Robust Controller 


The following controller is obtained from Zhang and Davison [14], and it can be shown to produce 

“perfect control” (i.e., the transient error in the system can be made arbitrarily small) in the system as 
€— 0: 

1 (s+1? @ 

~ €(es+1)2 5 


(Y — ret) (23.59) 


where 
1.7599 x 10° —3.4710x 10°  —1.0869 x 10% 


© := | —1.7599x 10° 3.4710 x 10° =—1.0870 x 10° (23.60) 
—3.9998 x 107 —3.0545x 104 —7.8258 x 10° 
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TABLE 23.7 Data Matrices for Distillation Column Model 


A= 
xl x2 x3 x4 x5 
xl —0.01400 0.00430 0 0 0 
x2 0.00950 —0.01380 0.00460 0 0 
x3 0 0.00950 —0.01410 0.00630 0 
x4 0 0 0.00950 —0.01580 0.01100 
x5 0 0 0 0.00950 —0.03120 
x6 0 0 0 0 0.02020 
x7 0 0 0 0 0 
x8 0 0 0 0 0 
x9 0 0 0 0 0 
x10 0 0 0 0 0 
xll 0.02550 0 0 0 0 
x6 x7 x8 x9 x10 
xl 0 0 0 0 0 
x2 0 0 0 0 0 
x3 0 0 0 0 0 
x4 0 0 0 0 0 
x5 0.01500 0 0 0 0 
x6 —0.03520 0.02200 0 0 0 
x7 0.02020 —0.04220 0.02800 0 0 
x8 0 0.02020 —0.04820 0.03700 0 
x9 0 0 0.02020 —0.05720 0.04200 
x10 0 0 0 0.02020 —0.04830 
xll 0 0 0 0 0.02550 
xll 
xl 0 
x2 0.00050 
x3 0.00020 
x4 0 
x5 0 
x6 0 
x7 0 
x8 0.00020 
x9 0.00050 
x10 0.00050 
xll —0.01850 
= 
ul u2 u3 
xl 0 0 0 
x2 5.00000e—06  —4.00000e—05 0.00250 
x3 2.00000e—06 —2.00000e—05 0.00500 
x4 1.00000e—06  —1.00000e—05 0.00500 
x5 0 0 0.00500 
x6 0 0 0.00500 
x7 —5.00000e—06 1.00000e—05 0.00500 
x8 —1.00000e—05 3.00000e—05 0.00500 
x9 —4,00000e—05 5.00000e—06 0.00250 


x10 —2.00000e—05 2.00000e—06 0.00250 
xll 0.00046 0.00046 0 
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TABLE 23.7 (continued) Data Matrices for Distillation Column 


Model 
C= 
xl x2 x3 x4 x 
yl 0 0 0 0 
y2 1.00000 0 0 0 
y3 0 0 0 0 
x6 x7 x8 x9 x10 
yl 0 0 0 0 1.00000 
y2 0 0 0 0 0 
y3 0 0 0 0 0 
x11 
yl 0 
y2 0 
y3 1.00000 
E= 
wl 
xl 0 
x2 0 
x3 0 
x4 0 
x5 0.01 
x6 0 
x7 0 
x8 0 
x9 0 
x10 0 
xll 0 


Properties of Closed-Loop System 


In order to determine the potential “speed of response” that may be obtained by the controller modeled 
by Equation 23.59, the following closed-loop eigenvalues of the system are obtained with « = 0.1 (see 
Table 23.9). 


TABLE 23.8 Properties of Distillation Column Model 


Open-Loop Eigenvalues Transmission Zeros 
—9.6031le—02 —9.1024e—02 
—7.0083e—02 —6.6830e—02 
—5.0545e—02 —5.0627e—02 
—1.2152e—04 —2.9083e—02 
—3.2355e—03 —2.2078e—02 
—3.3900e—02 —9.5168e—03 
—7.7594e—03 —6.4089e—03 
—1.9887e—02 

—1.8154e—02 

—2.4587e—02 


—1.4196e—02 
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Jy for Yreq(1) x 10° u for y,e1) 


¥ for Jre(2) ~ 109 U for Yret(2) 


0 0.5 1 1.5 2 0 0.5 1 15 2 


y for JpeA3) u for Vre(3) 


FIGURE 23.14 Response of closed-loop system for distillation column example using perfect robust servomecha- 
nism controller for unit step in y;e¢ given by yrep(1) = (100)', vreg(2) = (010); ree (3) = (001). 


Using the controller of Equation 23.59 with e = 0.1, the response of Figure 23.14 is then obtained for 
the case of a unit-step increase in yep = (10 0)’, (0 1 0)’, (0 0 1)’, respectively, with zero initial conditions. 
It is seen that “perfect control” indeed does take place; i.e., all transients have died down in less than 1 
second, the system displays “low interaction,” and no “excessive peaking occurs.” In real life, however, 
this controller would not be used because the control inputs are excessively large (see Figure 23.14). 

The following decentralized controller, obtained by the procedure given in [11], would be quite realistic 
to implement, however, since the control input signals do not “peak” now as they did in the previous 
controller. 


Robust Servomechanism Problem 


TABLE 23.9 Closed-Loop Eigenvalues 


—9.9102e+01 
—9.9102e+01 
—9.9097e+01 
—9.9097e+01 
—1.2776e+02 
—5.9778e+01 
—1.0030e+01 
—1.6633e+00 
—7.8470e—01 
—9.1245e—01 
—9.1245e—01 
—9.1207e—01 
—9.1207e—01 
—9.1027e—02 
—6.6832e—02 
—6.4090e—03 
—9.5169e—03 
—2.9083e—02 
—5.0626e—02 
—2.2078e—02 


23.4.3 Decentralized Robust Controller 


where Kj, K2, 3, K4, Ks are given in Table 23.10. 


u=Kiy+Kan+Ks3p 


N=) —Sret 
0=Ksp+Kay 


3.1595e+02i 


—3.1595e+02i 
3.1594e+02i 


—3.1594e+02i 


+2.8213e—Oli 
—2.8213e—O0li 
+2.8314e—Oli 


—2.8314e—01i 


TABLE 23.10 Decentralized Controller Gains Obtained 


Ky = 


1.7409e-+ 

0 

0 

4.5284e-+ 

0 

0 

5.7841e4 

0 

0 

1.0000e+ 

0 

0 
—1.1024e4 

0 

0 


+05 


+02 


+05 


+00 


+05 


0 

1.9140e4 

0 

0 

1.2143e4 

0 

0 

3.0466e- 

0 

0 

1.0000e+ 

0 

0 
—8.1558e- 

0 


+04 


+01 


+05 


+00 


+07 


—7.7460e- 
0 
0 
—3.7682e+ 
0 
0 
—1.0636e- 
0 
0 
1.0000e+ 
0 
0 
—1.3506e- 
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TABLE 23.11 Closed-Loop Eigenvalues 


—8.1558e+07 
—1.3506e+-04 
—1.1023e+03 
—1.7595e+00 4.1365e+01i 
—1.7595e+00 —4.1365e+01i 
—9.568le—03 +7.5529e—02i 
—9.568le—03 —7.5529e—02i 
—8.9273e—02 
—6.5720e—02 
—5.0556e—02 
—2.9058e—02 
—2.2081le—02 
—9.5034e—03 
—6.4076e—03 
—2.5926e—03 
—6.2808e—04 
—4.4157e—04 


Properties of Closed-Loop System 


The following closed-loop eigenvalues are obtained by applying the controller of Equation 23.61 to 
Equation 23.58 (see Table 23.11). 

Using the controller of Equation 23.61, the response of Figure 23.15 is then obtained for the case of a 
unit step in yrer = (1 0 0)’, (0 1 0)’, (0 0 1)’, respectively, and for the case of a unit step in the disturbance 
term w with zero initial conditions. It is seen that excellent tracking/regulation takes place; i.e., the output 
responses obtained show little interaction effects with no peaking occurring, and the control input signals 
are now quite realistic to implement. In addition, the controller has the additional advantage of being 
decentralized; i.e., the controller is particularly simple to implement. The time response of the closed-loop 
system, however, is now much slower compared to the case when perfect control is applied. 


23.5 Concluding Remarks 


In this overview on the RSP, the emphasis has been placed on the control of LTI continuous systems, 
and existence conditions and corresponding required controller construction to solve the RSP have been 
reviewed. To demonstrate the principles involved, various simple nonminimum- and minimum-phase 
examples were initially considered, and then a case study of a nontrivial system example was studied. 


23.6 Defining Terms 


Arbitrarily good approximate error regulation (AGAER): The property of a closed-loop system that 
permits arbitrarily good regulation to occur for arbitrary disturbance/tracking signals of a spec- 
ified class. 

Centralized fixed modes: Those modes of an LTI system that are not both simultaneously controllable 
and observable. 

Decentralized control: Refers to a controller in which the information flow between the inputs and 
outputs is constrained to be block diagonal. 
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beeen nrc astm 
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x 104 x 104 


FIGURE 23.15 Response of closed-loop system for distillation column example using decentralized robust 
servomechanism controller for unit-step in yrer given by yre¢(1) = (100)’, preg (2) = (010)'s Yrer(3) = (00 1)’ and 
for a unit-step disturbance. 
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Error coefficient: The steady-state error coefficient matrix associated with a closed-loop system for a 
given class of disturbance or reference input signals. 

Gain margin (GM): Given a stable closed-loop system, the GM (0, f) refers to the largest perturbation of 
gain in the system’s transfer function matrix that may occur before instability occurs. 
High-gain servomechanism controller (HGSC): A controller that gives perfect tracking for continuous 

minimum-phase systems. 

Minimum phase: A system whose transmission zeros are all contained in the open left complex half- 
plane. 

Nonminimum phase: A system that is not minimum phase. 

Perfect control: The ability of a controller to provide arbitrarily good transient response in the system. 

Perfect robust controller: A controller that solves the RSP such that perfect control occurs. 

Robust servomechanism problem (RSP): The problem of finding a controller to solve the servomecha- 
nism problem that has the property of providing exact asymptotic error regulation, independent 
of any perturbations in the plant that do not destabilize the system. 

Servomechanism problem: The problem of finding a controller to provide asymptotic error regulation 
and tracking for a system, subject to a specified class of disturbances and tracking signals. 

Servo-compensator: A compensator that is used in the construction of a controller to solve the RSP. 

Stabilizing compensator: A controller that stabilizes a system. 

Transmission zero: A generalization of the notion of a zero of a single-input/single-output system to 
multivariable systems. 
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Further Reading 


There are a number of important issues that have not yet been considered in this chapter. For example, 
when disturbances are measurable, so-called feedforward control [1,7] can be highly effective in minimiz- 
ing the effects of disturbances in the servomechanismproblem. 

In many classes of problems, the controller must often be constrained to be decentralized, e.g., in 
process control systems, power system problems, transportation system problems. A treatment of the 
so-called decentralized robust servomechanism problem, which arises in this case, is given in [8] and [11]. 

The effect of transportation delay in a system is often of critical importance in the design of controllers 
to solve the servomechanism problem. A treatment of systems that have time lag is given in [4]. 

Finally, it is often the case that no mathematical model is actually available to describe the plant that 
is to be controlled. In this case, if the plant is open-loop asymptotically stable, so-called tuning regulator 
theory [7] can be applied to obtain existence conditions and to design a controller, which can then be 
applied to the plant to solve the servomechanism problem. 

The above treatment of the servomechanism problem has been carried out in a state-space setting; 
alternative treatments may be found using other settings such as geometric methods, frequency-domain 
methods, polynomial matrix representation methods, coprime matrix factorization methods, etc. 
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Optimization problems and control problems are highly intertwined. If a control configuration has been 
decided upon, controller parameters or control input signals can be interpreted as decision variables of 
an optimization problem that reflects the desired specifications and constraints of the controlled system. 

In recent years, linear matrix inequalities (LMIs) have emerged as a powerful tool for approaching 
control problems that appear difficult ifnot impossible to solve in an analytic fashion. Although the history 
of LMIs goes back to the 1940s, with a major emphasis on their role in control in the 1960s through the 
work of Kalman, Yakubovich, Popov, and Willems, only during the last decades have powerful numerical 
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interior point techniques been developed to solve LMIs in a practical and efficient manner (Nesterov, 
Nemirovskii). Today, several commercial and noncommercial software packages are available that allow 
for simple coding of general LMI problems. For example, Yalmip [12] is a very flexible and noncommercial 
toolbox for defining and solving advanced optimization problems. 

Boosted by the availability of fast LMI solvers, research in robust control theory has experienced a 
significant paradigm shift. Instead of arriving at an analytical solution of an optimal control problem and 
implementing such a solution in software so as to synthesize optimal controllers, today a substantial body 
of research is devoted to reformulating a control problem to the question of whether a specific LMI is 
solvable or, alternatively, to optimizing functionals over LMI constraints. It is the purpose of this chapter 
to highlight the main role and use of LMIs in solving a large variety of control and estimation problems. 


24.1.1 Notation 


R and C denote the fields of real and complex numbers. Sets of real and complex matrices of dimension 
m x n are denoted by R™*" and C”*". A matrix A € C”*” is Hermitian if it is square (m = n) and if 
A= A*, where the star denotes taking complex conjugate transpose. For real matrices, this amounts to 
saying that A= A! in which case A is said to be symmetric. The vector spaces of all n x n Hermitian 
and symmetric matrices will be denoted by H!” and S", respectively, and we will omit superscript n if the 
dimension is not relevant for the context. A Hermitian or symmetric matrix A is called negative definite, 
negative semidefinite, positive definite, or positive semidefinite if x* Ax < 0, < 0, > O0or> 0 forall nonzero 
complex vectors x € C”. We will denote this by A < 0, A x 0, A > 0, and A > 0, and extend this notation 
to expressions A = B to mean that A — B = 0 for any A, B € H”. A congruence transformation of a square 
matrix M is a mapping M +> T'MT with some nonsingular T. The operator col(-) stacks its arguments 
in a vector, as in col(a, b) = ( b)> where a and b are vectors or matrices with the same number of columns. 


-£," denotes the space of all signals x : [0,0o) — R” with finite energy ||x||_% :=, are \|x(t)||? dt, where 


||.|| is the Euclidean vector norm. We refer to the appendix for a brief summary on notions of convex sets 
and convex functions. 


24.2 LMIs and Convexity 


An LMI is a constraint of the form 


F(x) := Fo +x, Fy +--+ +xnFy < 0, (24.1) 


where Fo, F,...,F, are given real symmetric matrices and where x = col(x1,...,X,) is a vector of 
unknown real scalar decision variables. 

The inequality F(x) < 0 means that x should render the symmetric matrix F(x) negative definite, that 
is, the maximum eigenvalue of F(x) should be negative. The LMI (Equation 24.1) gives rise to two types 
of questions: 


1. The LMI feasibility problem amounts to testing whether there exist real variables x),..., x, such 
that Equation 24.1 holds. 

2. The LMI optimization problem amounts to minimizing a cost function c(x) = cx ++ +++ CyXp 
over all x1,...,Xn that satisfy the constraint (Equation 24.1). 


Classical linear programs easily fit in this formalism, but also quadratic programs and some instances of 
convex quadratically constrained quadratic programs can be reformulated in this setting. In fact, the LMI 
optimization problem is a natural generalization of a linear program in which inequalities are defined by 
the cone of positive definite matrices. 
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In most control applications, LMIs arise with matrix variables rather than vector variables. This means 
that we consider inequalities of the more general form 


F(X) < 0, 


in which X is a matrix that belongs to an arbitrary finite-dimensional vector space V of matrices and 
where F : ¥ > S" isan affine function. We recall that affine functions assume the form F(x) = Fo + T(x), 
where Fo is fixed and where T is a linear map. Affine functions are, therefore, linear mappings plus some 
offset. For this reason, an LMI is actually better called an affine matrix inequality, but the world has 
decided to accept this erroneous nomenclature. 


Example 24.1 


A simple example of an LMI in the matrix valued unknown X = X/ is 

F(X) = A'X +XA+Q <0, 
where A and Q=Q! are given square real matrices. This is a special case of Equation 24.1 by 
expanding X as X = pie xXx, where Xz =X), k=1,...,n, is a basis of the set of symmetric 
matrices ¥. Indeed, F is affine and F(X) = F()°y_, xXx) = Q+ Dop_y Xk (A'X, +X;A) which is of the 
form as in Equation 24.1 with Fg = Qand Fy = A'X, + X;,A. 


Of course, one is led to wonder what the practical interest in studying constraints of the special form 
as in Equation 24.1 might be. There are a number of answers to this question. First, Equation 24.1 defines 
a convex constraint on the decision variable. This means that the solution set S := {x | F(x) ~ 0} of the 
LMI F(x) ~< 0 is convex. We refer the reader to the appendix for a brief overview of notions and results 
on convex sets and convex functions. Although the convex constraint F(x) < 0 on x may seem rather 
special, many convex sets can be represented in this way and, in fact, have more attractive properties than 
general convex sets. Second, the solution set of every finite set of LMIs 


Fi (x) ~ 0,..., Fy(x) <0 


can again be represented as one single LMI F(x) < 0 by setting F(x) = diag(F\(x),..., Fy(x)). Hence, 
multiple LMI constraints on a decision variable x is again an LMI constraint. Third, and very importantly, 


the partitioned LMI 
Fy (x) ao) 
F(x) = 0 
@) ee Fop(x)) ~ 


is equivalent to 


Fy1(x) <0 
So2(x) == Fo2(x) — Foi (x)Fii (x)! Fi2(x) < 0 


and at the same time equivalent to 


S14 (x) = Fyy (x) — Fyo(x)Fo2(x)~! Foi (x) < 0. 
Fy2(x) <0 


Since F is affine, this equivalence means that also specific types of quadratic and rational inequalities 
can be reformulated as Equation 24.1. The expressions S1;(x) and S22(x) are called Schur complements of 
Fy2(x) and Fj; (x) in F(x). The above equivalences follow from the fact that congruence transformations 
of symmetric matrices leave the number of positive and negative eigenvalues invariant. In particular, 
M <0, if and only if T'MT < 0 for any nonsingular matrix T. For example, the first follows by taking 


M = F(x) and T = (i —Fix (a) FaG) : 
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24.3 Numerical Solutions of LMIs 


Optimization problems over symmetric semidefinite matrix constraints belong to the realm of semidef- 
inite programming or semidefinite optimization. Although we mainly focus on control problems here, 
semidefinite programs occur in combinatorial optimization, polynomial optimization, topology opti- 
mization, and so on. In the last few decades, this research field has witnessed incredible breakthroughs in 
numerical tools, commercial and noncommercial software developments, and fast solution algorithms. 
In particular, the introduction of powerful interior point methods allow us to effectively decide about the 
feasibility of semidefinite programs and to determine their solutions. 

This section aims to indicate briefly the main ideas behind interior point solvers of convex optimization 
programs. Let the solution set S := {x € R” | F(x) < 0} of the LMI (Equation 24.1) be the domain of a 
convex function f : S > R which we wish to minimize. That is, we consider the convex optimization 
problem 


Vopt = inf f(x). (24.2) 


The idea behind interior point methods is to solve this constrained optimization problem by a sequence 
of unconstrained optimization problems. For this purpose, a barrier function o is introduced. This is a 
function $ : R” — R which is required to 


1. be strictly convex on the interior of S and 
2. approach +oo along any sequence of points {x,}°°, in the interior of S that converges to a 
boundary point of S. 


Given a barrier function 6, the constraint optimization problem is replaced by the unconstrained opti- 
mization problem to minimize the functional 


filx) =f (x) + to(x), (24.3) 


where t > 0 is a penalty parameter. The main idea is to determine a curve t +> x(t) that associates with 
any t > 0 a minimizer x(t) of f;. The minimum of f; is attained in the interior of S. Subsequently, the 
behavior of this mapping is considered as the penalty parameter t decreases to zero. In almost all interior 
point methods, the unconstrained optimization problem is solved with the classical Newton-Raphson 
iteration technique to approximately determine a local minimizer of f;. Since f; is strictly convex on R", 
every local minimizer of f; is guaranteed to be the unique global minimizer. Under mild assumptions 
and for a suitably defined sequence of penalty parameters t,,, t, > 0 as n — ov, the sequence x(t,) will 
converge to a point x*. That is, the limit x* := limy—oo x(tn) exists and vopt =f (x*). If, in addition, x* 
belongs to the interior of S, then it is an optimal solution to Equation 24.2; otherwise an almost optimal 
solution of Equation 24.2 can be deduced from the sequence x(t,). 

Interior point methods can be applied to either of the two LMI problems defined in the previous 
section. If we consider the feasibility problem associated with the LMI F(x) < 0, then (f does not play a 
role and) one candidate barrier function is the logarithmic function 


(x) = 


log det(—F(x)~!)  ifx eS, 
otherwise. 
If S is bounded and nonempty, ¢ will be strictly convex. This implies the existence of a unique xopt such 
that (opt) is the global minimum of >. The point xopt belongs to S and is called the analytic center of 
the feasibility set S. 
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The LMI optimization problem to minimize c(x) subject to the LMI F(x) < 0 can be viewed as a 
feasibility problem for the LMI 
G(x) := te =! p ) <0, 


(0) F(x) 
where t > to := infyes c(x) is a penalty parameter. Using the same barrier function yields the uncon- 


strained optimization problem to minimize 


1 
t —c(x) 
for a sequence of decreasing positive values of t. Due to the strict convexity of g; the minimizer x(t) of g; 


g(x) := log det(—G;(x)~1) = log + log det(—F(x)7') 


is unique for all t > to. Since closed-form expressions for the gradient and Hessian of g; can be obtained, 
a Newton iteration is an efficient numerical method to find minimizers of g;. Currently, much research 
is devoted to further exploiting the structure of LMIs in order to tailor dedicated solvers for specific 
semidefinite programs. 


24.4 Stability Characterizations with LMIs 


Around 1890, Aleksandr Mikhailovich Lyapunov made a systematic study of the local expansion and 
contraction properties of motions of dynamical systems around an attractor. He worked out the idea that 
an invariant set of a differential equation attracts all nearby solutions if one can find a function that is 
bounded from below and decreases along all solutions outside the invariant set. Such functions are called 
Lyapunov functions and they have been used to prove stability of equilibria of differential equations ever 
since. 
Consider the differential equation 

x(t) = f (x(t), £) (24.4) 
with finite-dimensional state space X = R” and wheref : X x R — X is assumed to be sufficiently smooth 
so as to guarantee existence and uniqueness of the solution x(t, to, xo) that satisfies the initial condition 
x(to, to, Xo) = xo € X. The differential equation (24.4) is time-invariant, if solutions satisfy 


x(t + t, to + T, x0) = x(t, to, xo) 


for any t € R with t and t + t in the interval of existence. A point x* € X is called an equilibrium or fixed 
point of the differential equation if x(t, to, x*) = x* satisfies Equation 24.4 (which implies that f(x*, t) = 0 
for all t > fo). A wealth of stability concepts associated with fixed points of differential equations exists. 
Here, we focus on just one. 

The equilibrium point x* of Equation 24.4 is called exponentially stable if for all t) € R, positive numbers 
a, B, and 8 exist (all possibly depending on tp) such that 


xo — x" |] <8 => Ilx(t, t0,x0) — x" ll < Bllxo —x* |e 8" for all t > to. (24.5) 


If a, B, and & do not depend on fo, then x* is said to be uniformly exponentially stable. If 8 is arbitrary, 
x* is called globally exponentially stable. 

Hence, a fixed point is exponentially stable if all solutions of the differential equation that initiate 
nearby x* converge to x* with an exponential rate a > 0. The following result is standard and relates 
exponential stability of linear time-invariant differential equations to LMI feasibility. 


Theorem 24.1: 


Let A € R"*". The following statements are equivalent. 
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1. The origin is an exponentially stable equilibrium point of x = Ax. 
2. Alleigenvalues (A) of A belong to C™ := {s € C | It(s) < 0} (ie., A is Hurwitz). 
3. The LMIs A'X + XA < Oand X > Oare feasible. 


Any solution X of the LMIs in item (3) defines the quadratic function V(x) := Xx that serves as 
a Lyapunov function for the equilibrium point x* = 0 of the differential equation x = Ax. Indeed, V 
achieves its minimum at x* = 0 and its derivative in the direction of the vector field Ax is 


£ x(0)Xx(0 = X(t)'Xx(t) + x(t)'Xx(t) = x(t) [ATX + XA]x(t), 


and hence nonincreasing by item (3). Rather straightforward arguments lead to Equation 24.5, where 3 is 
arbitrary, B = \/kmax(X)/Amin(X), and a > 0 is any number for which A'X +XA+20X <0. 

For many applications in control and engineering one may be interested in characterizing eigenvalue 
locations of A in more general stability regions than C~. 

For a real symmetric matrix P € S*”, the set of complex numbers 


iar 6 
co={seci (5) P(5)<ol 
is called an LMI region. 


Important stability regions such as half-planes Cgtap 1 := {s | 2(s) < a}, circles Cota 2 = {5 | |s] <r}, 
or conic sectors Cetap 3 = {s | Jt(s) tan(8) < |S(s)|} can be represented by LMI regions £p,, £p,, and Lp,, 
respectively, by taking 


0 0 sin(®) —_cos(0) 

_ (2a 1 a2 -r? 0 = 0 0 —cos(§) — sin(@) 
aa ( a r= ({ O) = sin(@) —cos(0) 0 0 
cos(9) _ sin(@) 0 0 


LMI regions include sets bounded by circles, ellipses, strips, parabolas, and hyperbolas. Since any finite 
intersection of LMI regions is again an LMI region, one can virtually approximate any convex region in 
Cas long as it is symmetric with respect to the real axis. 

To present the main result of this section, we recall the definition of the Kronecker product A ®@ B of 
two matrices A € C”*", B € Ck**, which is the mk x n@ matrix 


AyB... = AyB 
AmB... AmnB 


The following result as originating from [4] is an interesting and elegant generalization of the stability 
characterization in Theorem 24.1. 


Theorem 24.2: 


Let A € R"*". The following statements are equivalent. 


1. All eigenvalues of A are contained in the LMI region 


Keer) (F a) (@) <4} 
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2. There exists X = X' such that 


I \*(X@Q X@S I 
X>0 and aS (xed X@R) \A@I <0. 


The condition in item (2) isan LMI in X. The result of Theorem 24.1 is recovered by taking Q = 0,S = 1, 
and R= 0. With Q = —1,S = 0, and R = 1, the LMI region corresponds to the open unit disc; hence A 
has all eigenvalues within the open unit disc iff there exists X > 0 such that A'XA — X < 0. In turn, this 
LMI test is equivalent to saying that the discrete-time system x(k + 1) = Ax(k) is exponentially stable. 


Example 24.2 


Consider the problem to find a stabilizing feedback law u = Fx that simultaneously stabilizes the 
systems xX = A,x + B,u, where k = 1,...,4. and 


w=(i Sh mG mG De 2=( 
w= 9) m=Q) AG 3) HG) 


By Theorem 24.1, the equivalent problem is to find X, > 0 and F such that (Ay + BgF)'X, +X, (Ay + 
B,F) < 0 fork = 1,...,4. Since both X, and F are unknown, this is not an LMI constraint. However, 
assuming X, =--- = X4 =X, a congruence transformation with the matrix Y := X—! transforms the 
five matrix inequalities to 


Y>0, AY +YA,+ByM+M'B, <0, k=1,...,4, 


where we set M = FY. These are LMIs in Y and M. When implemented with the given matrices, this 
set of LMIs turns out to be feasible and any solution defines a feedback F = MY—' that solves the 
stabilization problem. One of these feedbacks is computed to be F = (—4.3874 —10.6332). An 
analogous synthesis strategy can be applied for the simultaneous pole-placement problem which 
amounts to finding F such that eigenvalues of Ay + B,F belong to an LMI region Lp for all k. Its 
solution is then an application of Theorem 24.2. 


24.5 Performance Characterizations with LMIs 


In this section, we consider a linear system 
x= Ax+Bd, e=Cx+Dd, x(0)=0, (24.6) 


in which d is viewed as an undesired external disturbance and e¢ is an error output. Many control synthesis 
problems can be translated into a question of disturbance attenuation: the controller should reduce the 
effect of the disturbance d onto the error e as much as possible. In this section we quantify or analyze the 
effect of d onto e for the system (Equation 24.6) whose transfer function matrix is given by 


T(s) = C(sI —A)~'B+D. 


24.5.1 #4-Performance 


Let us assume for Equation 24.6 that A is Hurwitz and D = 0. If the number of inputs is m then B = 
(bi,..., bm) has m columns. If the system is excited with a unit impulse in the vth input, it responds with 
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the output trajectory z(t) = Ce4"b,. The energy of this output trajectory equals 
[o,0) Co T, 
i [Ce“* by ]"[Ce*' by] dt = bt ( i er" Clce ar) by = bLYoby (24.7) 
0 0 


with Yo being the observability Gramian, the unique solution of the Lyapunov equation 
ATY9 + YoA+C'C =0. (24.8) 


If we add the output energies for impulsive inputs in all input components we obtain 


m oO m 
om i z,(t)'z,(t) dt= > Trace(byb] Yo) = Trace(BB'Yo) = Trace(B'YoB), 
v=1 0 


v=1 


where Trace denotes the sum of the diagonal elements of any matrix. In view of the explicit formula 
for the observability Gramian as used in Equation 24.7 combined with Parseval’s theorem, we infer that 
Trace(B'YoB) actually equals 


CO 
Toe = = Trace [ [C(iml — A)~'B]*[C(iwl — A)~!B] dw. (24.9) 
1 —0o 

Note that ||T'||4 is the so-called #-norm of the transfer matrix T, the name of which is motivated by 
the theory of Hardy-spaces in pure mathematics; this relation is not relevant for our purposes. 

We have actually derived the impulse-response performance interpretation of the #4-norm. Moreover, 
this interpretation also provides a concise link to classical linear quadratic control, since the impulse 
response Z,(-) is identical to the output response if the system’s state is initialized as x(0) = by. 

Let us briefly touch upon the stochastic interpretation of the “4-norm. If w is white noise with unity 
covariance, the asymptotic variance of the output process of Equation 24.6 satisfies 


lim E[z(t)'z(t)] = Trace ( lim CElx(t)x(t)1C") — Trace(CXoC"), 
t>0o too 
where Xo is the system’s controllability Gramian, the unique solution of 


AXy + XoA'+ BB'=0. (24.10) 


Since T(s)' = B'(sf — A')~!C, we can conclude that the asymptotic output variance is equal to || T "||. 
Due to Equation 24.9 this is the same as ||T'||.~%, which leads to a dual version for computing ||T'|.4. 
Let us summarize our findings as follows: If D=0 and Xo and Yo are the system’s controllability and 
observability Gramians satisfying Equations 24.10 and 24.8, respectively, then || The = Trace(CXpC') = 
Trace(B'YoB) is the sum of the energies of the output of Equation 24.6 for impulsive inputs in each input 
channel, and it also equals the asymptotic output variance if the input is white noise with unity covariance. 

Note that .“4-norms can be easily determined by solving linear equations and computing traces. Since 
these explicit formulas are inadequate for applying the synthesis procedure as developed in the next 
section, let us provide a genuine characterization of a bound on the .#4-norm by LMIs. It is important to 
stress that this formulation actually combines an LMI characterization of system stability with a bound 
on system performance. 


Theorem 24.3: 


A is Hurwitz and ||T ||44, < y iff D=0 and there exist X = X! and W = W' with 


ae +XA XB ) @ C 
<0, 


Brix =i Cow >0 and Trace(W) <y. (24.11) 
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Sketch of proof of “if” 


For the general manipulation of performance specifications it is an instructive illustration of how to show 
that the feasibility of the system of LMIs (Equation 24.11) implies ||T||.4 < y. Indeed, by taking Schur 
complements we infer that Equation 24.11 leads to 


1 
A'X + XA+ —XBB'X <0, X>0, CX7~!C'<W, Trace(W) <y. 
Y 
The first inequality implies A'X + XA < 0. Together with X > 0 this implies (Theorem 24.1) that A is 
Hurwitz. On the other hand, X) = (yX)7! satisfies 
AX + XoA'+BB' <0 and Trace(CXC') < Trace(yW) < y?. 


Combining the latter inequalities with Equation 24.10, we get A(X —Xo)+ (Xo —Xo)A! < 0, which 
implies, using the stability of A, that Xp — Xo > 0. Consequently, also Trace(CX9C') < y?, which yields 
that ||T lg <y. 


24.5.2 #.-Performance 


A different way of quantifying the effect of the disturbance d on the output e in Equation 24.6 is in terms 
of the so-called energy gain 
lela 


ITs t= a 
0<\ldll_¢,<oo lldllz, 

The norm reflects the worst amplification of disturbances on outputs if measuring the sizes of the input 
and output signals in terms of their :-norm or energy. Dissipativity theory [19] provides a direct path 
toward an LMI characterization of an upper bound ||T|| 4, < y. 


Theorem 24.4: 


A is Hurwitz and ||T || a, < y iff there exists X > 0 with 


an T 
I 0\ (0 X\(I 0 Do EN ayer OV 60". F 

( s) (: ) G mile b) ( 0 i) @ ) a Gei2) 
Sketch of proof of “if” 
Let us assume that Equation 24.12 holds. Then the left-upper block of Equation 24.12 just reads as 
A'X +XA+C'C <0, which implies A'X + XA < 0. Therefore X > 0 guarantees that A is Hurwitz. Since 
the inequality Equation 24.12 is strict, it continues to hold if we replace —y” by —(y—«)* for some 
suitably small € > 0. Let us choose any d with 0 < ||d|| 4% < oo and let ¢ be the output of Equation 24.6. If 


we right-multiply the perturbed version of Equation 24.12 with col(x(t), d(t)) and left-multiply with its 
transpose, and if we exploit the relations in Equation 24.6, we obtain 


x(t)\' (0 X\ (x(t)\ . (d(t)\' (—(y—©21_ (0° (d(8) 
Ee G 0 ee (Cn ( 0 1 ee 
= 4 a Xe(0) + e(t)'e(t) — (y —©)*d(t)'d(t) < 0. 


dt 
By integration on [0, T] and with x(0) = 0 we infer 


T T 
x(T)'Xx(T) + / e(t)'e(t) dt < (y—€)” / d(t)'d(t) dt forall T > 0. 
0 0 
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Since both x(-) and x(-) are of finite energy, x(T) — 0 for T — oo. After taking the limit T — oo we 
hence obtain llellZ, <(y- €)"IIdll%, or |lel|l_a/Ildllz& < y —€. Since this holds for all 0 < ||d||.4 < co 
we finally arrive at || T || 4, < y. 

We started from the formulation of the LMIs (Equation 24.12), which gives the most insight for a 
dissipation-based proof of these results. In the literature, more common equivalent representations of the 
LMI (Equation 24.12) are 


AIX -EXA MRS OF 
)<o or (Schur) BX —-y7I D'}) «0. (24.13) 


Cs +XA+C'C XB+C'D 
C D -I 


B'X+D'C D'D-y?I 


Equivalently, A'X +XA+C'C+(XB+C'D)(y7I — D'D)71(B'X + D'C) <0, and D'D—y?I <0, 
which touches upon the relation to Riccati inequalities and equations. 


24.5.3 The Kalman-Yakubovich-Popov (KYP) Lemma 


We have established the link of Equation 24.12 to the time-domain energy-gain by dissipation arguments. 
The relation of this LMI to the frequency-domain is the subject of the celebrated KYP lemma. Algebraic 
arguments proceed as follows. If Equation 24.12 holds with X = X', then A'X + XA < 0 implies that A 
has no eigenvalues on the imaginary axis. For w € R let us observe that 


(im] — A)~1B\* (I 0\ (0 X\ (I 0)\ ((iwlI —A)7!B 
PG ae a CT?) 


= [(iwl — A)‘ B]* Gy ) ( ) Ge ) Gor-ay's 


= [(iw! — A)! B]*[iwX — iaX (inl — A)~'B=0. 


Therefore, feasibility of Equation 24.12 implies the validity of the frequency-domain inequality 


ay ee?) 
Gad ( . ') Gre: for all w € RU {ov}, 


which follows for w = oo from the right-lower block of Equation 24.12. Note that this inequality translates 
into T(iw)*T(iw) — y*I < 0 or Omax(T(iw)) < y for all wm € RU {oo}. In turn, this reveals the relation of 
the .“-gain bound to the peak of the largest singular value of the system’s frequency response, which is 
simply the classical .-norm for transfer matrices without poles on the imaginary axis, or the #.-norm 
for transfer matrices whose poles are all contained in the open left-half-plane. 

The following result captures a generalization of the strict version of the KYP lemma for an arbitrary 
symmetric matrix P and without involving any sign-constraint on X. 


Theorem 24.5: 


In the case that A has no eigenvalues on the imaginary axis then 


ae I 
(acs) P ice) <0 forall w € RU {ov}, (24.14) 


iff there exists some X = X' with 


7 Wy 
¢ 4 (: ) ({ +(e 5) al p) <° (24.15) 
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The survey article [10] provides a nice historical account of the development around the KYP lemma 
with many references, also to the Russian literature, and discusses further versions of this result even 
without any a priori hypotheses. In particular [2] is a rich source for the link of the KYP lemma to 
semidefinite programming duality and the related control theoretic interpretations. 


24.5.4 Variants 


The books [3,7] contain a whole variety of concrete variations on the theme of formulating performance 
specifications with LMIs. Let us provide a sample. 


24.5.4.1 Generalized #4-Performance 


If replacing Trace(W) < y by W < yl in Theorem 24.3, the formulated LMIs characterize that the system 
gain from finite energy input signals to the peak value of the output signal is bounded by y: 


ello 


7 <y with |lel|g, := sup lle(t)||. 
0<\ldl_z,<oo Idle, t>0 


This criterion allows to impose time-uniform bounds on the error variable under the assumption that the 
disturbance has bounded energy. 


24.5.4.2 Quadratic Performance 


Our discussion of -gain performance opens the path toward the following generalization. With a 
symmetric weighting matrix P, quadratic performance with index P is achieved if an € > 0 exists, such 
that the following integral quadratic constraint (IQC) on the performance channel d — e of the stable 
system in Equation 24.6 is satisfied: 


© /a(t)\", (a(t) F 
i es : Gs) dt <—elld|l’y, forall de 2. 


This time-domain specification directly translates into the frequency-domain inequality in Equation 24.14 
and, due to Theorem 24.5, into feasibility of the LMI in Equation 24.15. If the right-lower block of P is 
positive semidefinite, it is elementary to see that stability of A is guaranteed by imposing the additional 
positivity constraint X > 0. 


24.5.4.3 Discrete-Time 


All described results have counterparts for discrete-time systems 
x(t+1)=Ax(t)+Bd(t), e(t)=Cx(t)+Dd(t), x(0)=0, t=0,1,2,.... 


If A has all its eigenvalues in the unit disc (Schur stability), then quadratic performance holds, by definition, 
if there exists some € > 0 with 


[e.e) T [e.e) 
5 (i))'» (4) =F ava 
t=0 


t=0 


This is equivalent to the frequency-domain inequality 


|e ae 
(a) P(()) <9 forall zEC, |z|/=1 
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on the unit circle, which is, in turn, equivalent to the existence of X = X' satisfying the LMI 


bP DEX OVE O10 TN ale Ty oy 

A B 0 X/\A B C D C D ; 
If the right-lower block of P is positive semidefinite, Schur stability of A is once again guaranteed by X > 0. 
We stress the pleasing parallel structure of continuous-time and discrete-time performance formulations, 


which is even more striking for synthesis since the proposed general procedure applies to both domains 
without the need for any adaptation. 


24.6 Optimal Performance Synthesis 


Let us now consider a system 


x = Ax + Bod + Bu, 
e=Coxt+ Dod + Eu, (24.16) 
y =Cx+ Fd, 


where, in addition to the disturbance d and the error e, the signal u is a control input and y is a measured 
output. In feedback synthesis, the goal is to determine a controller 


Xe =Acxe + Bey, 
aie (24.17) 
uU=C.x%,+ Dey; 
which feeds the measurements y back to u such that the controlled system, the interconnection of 
Equations 24.16 and 24.17, is internally stable and satisfies a desired performance property (Figure 24.1). 
Note that the controlled system with state & = col(x, x,) is easily seen to be described by 


A+BD-C BC, | By) + BDF 
= B.C A. B.F . (24.18) 
Co+ED-C EC; | Do + ED,F 


E=AE+Bd | (4 5) 
with 
e=CE+Dd c|D 


In view of the generalized plant framework [18,20], it is essential to understand that this innocent 
problem formulation comprises surprisingly many specific configurations as they are needed in one- or 
two-degrees of freedom controller synthesis for reference tracking and disturbance attenuation. In this 
section, emphasis is put on an .#4.-norm bound as a measure of performance. 


System 


—| Controller 


FIGURE 24.1 Generalized plant configuration. 
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24.6.1 State-Feedback Synthesis 


The simplest control law is state-feedback u = D,x with a gain D,, which leads to the controlled system 


x = (A+ BD,)x + Bod, 


(24.19) 
e=(Co+ED,)x + Dod. 


Using Theorem 24.4 and with the LMI expressed as in Equation 24.13, this controller stabilizes the system 
in Equation 24.19 and renders the .#4.-norm of the transfer matrix d — e smaller than y iff there exists 
some X with 


(A+BD,)'X+X(A+BD,) XBo (Co+ED,)" 
X>0O and BUX -y"I Dj) <0. (24.20) 
(Co + ED;) Do —I 


Recall that the first inequality guarantees stability while the second captures performance. We observe that, 
in synthesis, we need to search for both D, and X in order to satisfy Equation 24.20. The performance 
inequality is, with some abuse of notation, a so-called bilinear matrix inequality problem since the 
left-hand side is affine in X (for fixed D,) and affine in D, (for fixed X). Actually, a large variety of design 
problems in control can be easily seen to admit this structure. Unfortunately, however, bilinear matrix 
inequalities are as hard to handle as general nonlinear programs. Fortunately, for the problem at hand 
there is a surprisingly simple remedy. 

There exists a celebrated procedure that actually turns the nonconvex bilinear matrix inequality 
(Equation 24.20) into a convex problem of LMIs. This is achieved by applying a nonlinear change of 
variables (D,, X) — (M, Y) and a congruence transformation to Equation 24.20. Indeed if we transform 
Equation 24.20 by congruence with X~! and diag(X~!, I, I), respectively, and if we introduce the new 
variables 

Sk) and... = D.x—, (24.21) 


we arrive at 


(AY+BM)'+(AY+BM) Bo (CoY+EM)" 
Y>0O and Bh -y7I D) <0. (24.22) 
(CyY + EM) Do —I 


Obviously the constraints in Equation 24.22 constitute LMIs in (M, Y) whose feasibility can be verified. If 
(M, Y) isa solution of Equation 24.22, we can solve Equation 24.21 for (D,, X) as X = Y~-landD. = MY~! 
and perform a congruence transformation of Equation 24.22 with Y~! and diag(Y~!, I, I) which leads 
back to Equation 24.20. This proves, with X being a certificate, that D, is indeed stabilizing and achieves 
the desired performance specification. On the other hand, if the inequalities in Equation 24.22 are not 
feasible, it is assured that no state-feedback gain can exist for which both these properties are satisfied. 


24.6.2 Output-Feedback Synthesis 


A general output-feedback controller leads to the controlled system in Equation 24.18. It achieves stability 
of A and ||C(sI — A)“!B+ Dil, <y iff there exists some ¥ with 


A'X+XA XB Cl 
X>0 and BX -yI D'] <0. (24.23) 
Cc D -!I 


Due to the affine dependence of (A, B, C, D) on the controller matrices (A;, B,, C;, D), this involves again 
a bilinear matrix inequality. 
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It is pleasing that one can identify, again, a convexifying controller parameter transformation [13,17]. 


If we partition 
fv ile 
X= (fr ®) and X= (7 ‘ (24.24) 


according to A (where X and Y share their dimension with A and the *’s denote matrices that are 
irrelevant for our purposes) it reads as 


Rod XAY 0 U- XB\ (Ap. BV VV 10 
= F 24.25 
& a) ( 0 +5 sae aalee I (2420) 
We view this as a transformation (1, A,, B,, C,, D,) > (X, Y, K, L,M, N) =: v. This definition is moti- 
vated by the following easily verified relations: 


with VY := (yr 0 1x 


ss ) we have y vy = C =: X(v) and (24.26) 


AY+BM A+BNC |Bo + BNF 

T, T, 

( ee Y we ) 2 K  XA+LC|XBy)+LF | =: (anne . ] (24.27) 
CoY +EM Cy +ENC|Dp + ENF 

Although looking intricate, the key is the affine dependence of the blocks X(v), A(v), B(v), C(v), and 


D(v) on the new variables v. If Y is nonsingular, a congruence transformation with Y and diag(), I, I) on 
Equation 24.23 leads to 


V(ATX)V+YV(XA)Y Y'(xB) ylcr 
yixy>0 and (B'xX)y -yI oD! | <0, (24.28) 
cy D —I 
which is, in turn, nothing but the following LMI in v: 
A(v)'+A(v) Biv) C(v)P 
X(v)>0O and Biv)" —y7I D(v)'] <0. (24.29) 
C(v) D(v) —I 


This brings us to the following result which is true without any hypothesis on Y. 


Theorem 24.6: 


There exists a controller as in Equation 24.17 which stabilizes the system in Equation 24.18 and renders the 
KHoo-norm of the transfer matrix d — e smaller than y iff the LMI’s in Equation 24.29 are feasible. 


The actual design of a controller proceeds as follows. Find a solution of the LMIs in Equation 24.29; 
due to the first inequality in Equation 24.29 the matrix J — XY is nonsingular (Schur); therefore, one can 
find square and nonsingular matrices U and V with I— XY = UV'; then we can solve Equation 24.25 
for (A;, B;, C;, D,). This controller does the job since we can define the nonsingular matrix Y and solve 
for ¥ in Equation 24.26, which implies that Equation 24.27 is valid; then Equation 24.29 is nothing but 
Equation 24.28; since ) is nonsingular, this transforms into Equation 24.23 by congruence. 

In summary, we have reduced the design problem to find a stabilizing controller that establishes a 
bound on the .# -norm of the closed-loop system to an equivalent problem that amounts to checking 
the feasibility of the LMIs in Equation 24.29. Since y? enters these constraints in an affine fashion, 
one can minimize y? subject to the feasibility of these LMIs in order to directly compute the optimal 
Hoo -attenuation level that is achievable by stabilizing controllers. 
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24.6.3 General Synthesis Procedure 


Although we discussed -% -synthesis in quite some detail, we can extract the following generic procedure 
for moving from analysis to synthesis inequalities for a whole variety of other performance specifications 
that can be expressed by LMIs. 


¢ Rewrite the analysis inequalities in terms of the blocks V, XA, ¥B,C, and D. 

¢ Find a formal congruence transformation involving Y which leads to inequalities in terms of the 
blocks V'XY, V'(XA)Y, Y'(XB), CY, and D. 

¢ Then the synthesis inequalities are obtained by the substitution 


(XA) a _ Go an 
C D Civ) Div) 
with Equation 24.27 for v := (X, Y, K, L, M,N). 
¢ For state-feedback synthesis one can apply the very same procedure with the formulas X(v) = 
Y, A(v) = AY + BM, B(v) = Bo, C(v) = CoY + EM, D(v) = Do for v= (M, Y). 
¢ The controller construction is independent from the particular analysis inequalities and remains, 
both for state-feedback and output-feedback synthesis, unaltered. 


ylay > X(v), ( 


This procedure applies to .#4-synthesis as well as to the variants of the LMI analysis specification 
discussed in Section 24.5.4. As an illustration, the discrete-time system x(t + 1) = Ax(t)+ Bd(t), e(t) = 
Cx(t) + Dd(t) is Schur-stable and its 1,-gain is bounded by y iff there exists some V with 


T T 
I 0\ (-x 0)\/I 0 0 I\ (-y7I 0\/0 I 
eed € Bet aie eG aye 0 aie p) <2 
Since these can also be expressed as 
z 
x 0 A B xX 0\/\(A B 
eich & vote @) G att 5) ro 

a Schur complement argument reveals that these analysis inequalities can be written as 


xX 0 lr fal 


0 yr} BY DI j 
XA XB| X 0 
Gc "DP 0 I 


and thus admit the precise format in order to apply the general synthesis procedure. 

Based on the generic dualization and elimination results described in [16], it is often possible to 
eliminate matrix variables that only appear in one of these synthesis inequalities, with the benefit of 
reducing computational complexity. For example, one can eliminate all matrices K, L, M, and N from the 
Hoo -synthesis inequalities in order to arrive at the inequalities as proposed in [8,11]. 

Finally, in many practical problems the disturbance and error signals are partitioned as d= 
col(d),...,dp) and e = col(e),..., eq) in order to impose multiple individual performance requirements 
on some of the channels d, — e,, possibly with different norms. The general controller synthesis pro- 
cedure described in this section is applicable for this kind of multiobjective control problems provided 
that one is willing to allow some level of conservatism by expressing the combined desired specifications 
with one and the same matrix ¥ in the closed-loop system. This variant of multiobjective control, which 
is addressed as a Lyapunov-shaping paradigm in [17] in more detail, even allows the incorporation of 
pole-placement requirements on A in terms of general LMI regions. We refer to [17] for an instructive 
controller design example. The introduction of slack-variables offers a possibility to reduce conservatism 
as shown for discrete-time systems in [5], while an LMI solution of the general multiobjective control 
problem based on the Youla—Kucera parameterization of all stabilizing controllers is discussed in [15] 
and its references. 
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24.6.4 Observer and Estimator Synthesis 


Estimation problems are related to the configuration in Figure 24.2. Based on measurements y, the goal 
is to determine an estimator whose output approximates the system output z (which can be equal to the 
state or comprise components of d) as closely as possible, despite the corruption of the system by the 
disturbance d. If d is white noise with unity covariance, minimization of the asymptotic variance of e 
amounts to minimizing the .~4-norm of d — e which results in the classical Kalman filter. Alternatively, 
using the energy-gain of d — e as a performance indicator amounts to considering the “ -estimation 
problem. These and many other variants of estimation problems can be rephrased as an output-feedback 
synthesis problem for a system as described by Equation 24.16 with B = 0 and E = —I (which means that 
u does not excite the system dynamics). Let us stress that in this reformulation u admits the interpretation 
of the estimator output and e is the estimation error. 
If the estimator admits the structure of an observer with a to-be-designed gain D,, 


Xe = Axe +D-(Cxe—y), uU= Coxe; (24.30) 


then the dynamics of the state-error & = x — x; is described by 


& =(A+D,C)§+(Bo+D,F)d, e=Co&+Dod 
and defines the transfer matrix d — e. This representation is dual (transposed) to what we considered 
for state-feedback synthesis, and only slight modifications are required in order to determine the LMIs 
for synthesizing observer gains D, that stabilize the error dynamics and achieve, for example, an o- or 
44,-norm bound y ond > e. 

Let us now assume that A in Equation 24.16 is stable. Instead of assuming a particular structure, we 
can then try to find a general estimator Equation 24.17 with stable A, such that a desired performance 
level for d — e is achieved. In view of Equation 24.18 and the fact that B = 0, stability of A, now boils 
down to stability of A, and the very same output-feedback synthesis procedure can be followed in order 
to design optimal estimators by LMIs. 

However, it is interesting to observe that the structural property B = 0 allows a simplification of the 
convexifying controller parameter transformation which will be essential for robust estimator synthesis 
[9]. Indeed, starting from Equation 24.24 and with Z = Y~! let us define 


KL). 2 fU. OV fAe BN(VZ. 0 wane de 0 
e - a ') fe eal 0 ) ee (vt, HE Geen) 


After a direct computation we obtain 


ZZ 
ZX 


ZA ZA ZBo 
XA+LC+K XA+LC|XBo+LF =( 
Co + ENC + EM Co + ENC|Do + ENF 


yaya ( ) =: X(v) and (24.32) 


epee) = 


A(v)| B(v) 
cy | D 


C(v)|D(v) 
with an affine dependence on X, Z, K, L, M, and N. Therefore the general synthesis procedure in Section 


24.6.3 does apply, with the only modification being the use of these new substitution formulas and by 
recalling the relation Z = Y~! for the actual design of the estimator. 


—¢ 
- System q 
u mea 


Estimator 


FIGURE 24.2 Estimator synthesis. 
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24.7 Polytopic Uncertainties and Robustness Analysis 


First principle models of physical systems are often represented by state-space descriptions in which the 
various components of the state represent well-defined physical quantities. Variations, perturbations, or 
uncertainties in physical parameters lead to uncertainty in the model. Often, this uncertainty is reflected 
by variations in well-distinguished parameters or coefficients in the model, while, in addition, the nature 
and/or range of the uncertain parameters may be known, or partially known. Since very small parameter 
variations may have a major impact on the dynamics of a system, it is of evident importance to analyze 
parametric uncertainties of dynamical systems. Suppose that § = (8),...,8p) is the vector that expresses 
the ensemble of all uncertain quantities in a given dynamical system. Then there are at least two distinct 
cases that are of independent interest: 


a. Time-invariant parametric uncertainties: the vector 4 is a fixed but unknown element of an uncer- 
tainty set § C RP. 

b. Time-varying parametric uncertainties: the vector § is an unknown time-varying function 5 : R > 
R? whose values 8(f) belong to an uncertainty set 8 C R?, and possibly satisfy additional constraints. 


24.7.1 Time-Invariant Parametric Uncertainty 


Consider the uncertain time-invariant system defined by 
x = A(8)x, (24.33) 


where A(-) is a continuous function of the real-valued parameter vector § = col(3j,...,8p) which is 
only known to be contained in an uncertainty set 8 C R?. The problem of robust stability amounts to 
characterizing whether the equilibrium point x* = 0 of Equation 24.33 is exponentially stable for all 
parameters § € 8. With time-invariant uncertainties, Equation 24.33 is robustly stable iff A(8) is Hurwitz 
for all 8 € 8. Since 8 generally consists of infinitely many points, the verification of this condition is rather 
troublesome from a computational point of view. 

The uncertain system (Equation 24.33) is called quadratically stable if there exists X = X' such that 


X>0, A(8)'X+XA(8) <0 forall 868. (24.34) 


The importance of this definition becomes apparent after observing that V(x) := x'Xx is a quadratic 
Lyapunov function for Equation 24.33 which, by Theorem 24.1, implies that A(8) is Hurwitz for all 8 € 8. 
Hence, quadratic stability implies the origin of Equation 24.33 to be robust exponentially stable against 
time-invariant uncertainties 5 € 8. Unless 8 has a finite number of points, Equation 24.34 cannot be 
verified easily. Therefore, the following result is of considerable interest. 


Theorem 24.7: 


If A(8) is affine in 8 and 8 = co{8!,...,8“} then Equation 24.33 is quadratically stable if and only if there 
exists some X such that 
X>0, A(8*)'X+XA(8") <0, k=1,...,N. 


Hence, for polytopic uncertainty sets and affine parametric dependence in Equation 24.33, quadratic 
stability can be numerically verified by a feasibility test for a finite number of LMIs only. The proof 
is an illustrative example of the use of convexity. It requires showing that F(8) := A(8)'X + XA(8) < 0 
for all 8 € 8 if F(8*) <0 for k=1,...,N. To see this, first observe that F(-) is a convex function on 8 
whenever A(-) is affine. Second, any § € 8 can be written as a convex combination of the points 8!,...,8“ 


> 
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say 8 = yy a,5* with nonnegative coefficients a, that sum up to 1; hence F(8) = F Oo aS*) < 
Bie a. F(8*) < 0, which gives the result. 


24.7.2 Time-Dependent Parametric Uncertainty 


Robust stability against time-varying uncertainties is generally a more demanding requirement than 
robust stability against time-invariant uncertainties. Consider the system 


x(t) = A(8(4))x(E), (24.35) 


which is affected by an uncertain parameter curve § : R > 8. Unlike the case with time-invariant uncer- 
tainties, robust stability of the origin is now not implied by the condition that A(8) is Hurwitz for all 8 € 8. 
However, the uncertain system with time-varying parametric uncertainties is exponentially stable if there 
exists X > 0 such that Equation 24.34 holds. Therefore, quadratic stability does, in fact, imply robust 
stability against arbitrary fast time-varying parametric uncertainties. This is a nice, but in general conser- 
vative, test if additional a priori information on the uncertainty is available. For example, the parameter 
curves 8(-) are often known to be continuously differentiable and constrained in terms of their values and 
their rate-of-variation as 


s(t)e8 J&(t)ep forall teR. (24.36) 


Less conservative robust stability tests can be inferred by postulating the existence of parameter-dependent 
Lyapunov functions. A popular instance of such functions takes the form V(x, 8) := x'X(8)x and requires 
a search over matrix functions X(8) = X(8)' with 8 € 8. For notational convenience, let us introduce, for 
a continuously differentiable matrix function X(8), the “derivative” 


P 
OX(8,p) = } > OX (8)px, (8,9) € 8X p, (24.37) 
k=1 


where 0,X(-) denotes the partial derivative of the function X(-) with respect to the kth entry of 8 and 
where p; is the Ath component of the vector p. (We stress that 0X (8, e) is purely a symbolic notation which 
is not to be confused with the partial derivative of X(-) itself.) The following result provides a sufficient 
condition for robust stability and actually covers many tests in the literature. The proof provides much 
insight into the understanding of stability arguments based on parameter-dependent Lyapunov functions. 


Theorem 24.8: 


Suppose that 8 and p are compact subsets of R? and suppose that X(8) = X(8)' is a continuously differen- 
tiable matrix function that satisfies 


X(8)>0, AX(8, p) +.A(3)'X(8) + X(8)A(8) < 0 (24.38) 


for all8 <8 and p€ p. Then the origin of Equation 24.35 is exponentially stable for all time-varying 
parametric uncertainties that satisfy Equation 24.36. 


Proof. Suppose that X(8) satisfies Equation 24.38. Continuity of X(-) and compactness of § and p 
guarantee the existence of constants a, b, c > 0 such that, for all 8 c 8 and p € p, 


al < X(8)<bI, AX(8,p) + A(8)'X(8) +.X(8)A(8) < —cl. (24.39) 
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Let 8(-) and x(-) be a parameter curve and a state trajectory that satisfy Equations 24.36 and 24.35, 
respectively. With &(t) := x(t)'X(8(t))x(t) we clearly have 


P 
E(t) = x(t)" bs Axton x(t) + x(t) "TA(8(#))'X(8(t) + X(B()A(B(E)) Ix(0). 


k=1 


If we exploit Equation 24.39 we obtain al|x(t)||? < &(t) < bl|x(t)||? and E(t) < —c||x(t)||?. This implies 
that &(t) < —&(t) and hence &(f) < &(to) exp(—¢(t — fo)) for all t > fo. In turn, this leads to Ilx(t)||? < 
2 Ix(to)||2e7 6), which is Equation 24.5 for @ = c/(2b) and B = Jb/a. 

The constraints in Equation 24.38 on X(-) define a purely algebraic test that do not involve the system- 
or parameter-trajectories. The test is not easy to apply directly because the matrix function X(-) needs to 
satisfy a partial differential LMI. By considering specific classes of matrix functions X(-), Equation 24.38 
can be converted and implemented with LMI solvers. One of these classes is the set of affine symmetric 
matrix functions X :§ > S". 

A few special instances are worth mentioning. If the parameters are time-invariant, we have p = {0} 
and Equation 24.38 simplifies to the conditions X(8) > 0 and A(8)'X(8) + X(8)A(8) < 0. In that case, 
Equation 24.38 is also necessary for robust stability. If parameters vary arbitrarily fast, p is unbounded 
and Equation 24.38 is feasible only if the partial derivatives 0,X(8) vanish identically. This means that 
X(8) = X is not depending on 8 and we recover the quadratic stability test. 


24.7.3 Robust Performance 


In the previous subsections we have shown how tests for robust stability against parametric uncertainties 
can be inferred from characterizations of nominal stability. The same generalization applies in order to 
obtain conditions for verifying robust performance. Consider the uncertain parameter depending system 


x(t) = A(8(t))x(t) + B(S(t))d(t), (24,40) 

e(t) = C(8(t))x(t) + D(d(t))d(t), , 
where 8(-) is a continuously differentiable rate-bounded uncertainty that satisfies Equation 24.36. In the 
case of #4.-performance, we quantified the effect of d on e in terms of the YZ gain of the system. However, 
the output e of Equation 24.40 not only depends on the input d but also on the uncertainty 8(-). Hence 
we say that the robust Ly-gain is smaller than y if 


¢ for d=0 and for all parameter curves 8(-) satisfying Equation 24.36, x* = 0 is an exponentially 
stable equilibrium of the system in Equation 24.40. 
¢ for x(0) = 0 it holds that 


llellZ 
sup 


8(-) satisfies Equation 24.36 0<|dll ev, <oo I d||z, 


With some abuse of terminology, robust 4-gain performance is often referred to as robust H%0- 
performance, but it is important to realize that frequency-domain characterizations do not make sense 
when considering time-dependent parametric uncertainties in Equation 24.40. Only with time-invariant 
parameter uncertainties (p = {0}) does a robust “4-gain that is smaller than y imply that || Ts ||, < y for 
all 8 € 8, where 7; is the transfer function associated with Equation 24.40 for time-invariant parameters 
(t) = 8. The following result generalizes Theorem 24.4 to robust 22-gain performance. 
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Theorem 24.9: 


Suppose there exists a continuously differentiable matrix function X(8) = X(8)' such that X(8) > 0 and 


OX (8, p) + A(S)'X(8) + X(8)A(8) X(3)B(8) 
B(8)'X(8) 0 


ef 8 oy es ant a : 
C(3) D(8) 0 1) \c(s) D(s)) ~ 


for all’ € 8 and p € p. Then the uncertain system in Equation 24.40 has a robust L-gain smaller than y. 


(24.41) 


The proof of this result is analogous to that of Theorem 24.4. The main merit of Theorem 24.9 is that 
it converts robust .%-gain performance to an algebraic property. As in Theorem 24.8, the condition in 
Equation 24.41 requires the numerical search for a matrix function X(-) that needs to satisfy a partial 
differential LMI. Moreover, the discussion about the extreme cases concerning time-invariant or arbitrary 
fast time-varying parameters in Section 24.7.2 remains valid for Equation 24.41. Finally, let us remark 
that Theorem 24.9 extends, mutatis mutandis, to all other performance specifications that have been 
mentioned in Section 24.5. 


24.8 Robust State-Feedback and Estimator Synthesis 


Consider the system 


x(t) = A(S(t))x(t) + Bo(8(t))d(t) + BOS(t))u(t), 
e(t) = Co(8(t))x(t) + Do(8(t))d(t) + E(8(t)) u(t), (24.42) 
y(t) = C(8(t))x(t) + F(8(t))d (2); 


whose describing matrices are affected by a time-dependent parametric uncertainty 5(t) € R?. Let us also 
assume that the dependence of the system matrices is actually affine, and that 8(t) takes values that are, 
for t > 0, confined to the polytope 
8:=cof{d!,...,8N} CRP. (24.43) 

Robust controller synthesis deals with the problem of determining a feedback controller that processes 
measurements y to control inputs u, so as to guarantee robust stability and a desired robust performance 
specification on the mapping from the disturbance d to the output e of the controlled system. 

The robust state-feedback synthesis problem is a special case in which the whole state is assumed to be 
measurable (y = x in Equation 24.42) and the controller is a static feedback law u = D,x with some gain 
D,. In that case, the resulting closed-loop system is described by 


x(t) = [A(8(t)) + B(S(t)) De l(t) + Bo(S(t))d(t), 
e(t) = [Co(8(t)) + E(8(t))De]x(t) + Do(8(t))a(t). 
If applying Theorem 24.9 for a parameter-independent X(5) = X and exploiting affine parameter- 


dependence as for Theorem 24.7, we infer that the robust %-gain of the controlled system is smaller than 
y if there exists X = X! with 


[A(8*) + B(8*)D,J'X + X[A(8*) + B(S*)D.]  XBo(8*)  [Co(8*) + E(8*)D-]" 
X>0, Bo(8k)'X —y?I Do(3*)" <0 
Co(8*) + E(8*)D, Do(8*) —I 


for all k=1,...,N. (24.44) 
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Literally following the nominal synthesis procedure of Section 24.6.1 with the convexifying transformation 
in Equation 24.21 we infer that there exist (D,, X) satisfying Equation 24.44 iff there exist (M, Y) satisfying 


[A(S*)Y + B(S*)M]' + [A(8*)Y + BOS*)M] = Bo(8*)  [Co(8*)¥ + E(8*)M]" 
Y>0, Bo(8*)" —y?I Do(8*)T <0 
Co(3*)¥ + E(8*)M Do(3*) a 


forall k=1,...,N. 


This LMI feasibility problem in (M, Y) can be readily solved. If (M, Y) is a solution, the state-feedback 
gain D, = MY! guarantees a robust -4)-gain that is smaller than y for the closed-loop system. 

This procedure works smoothly because the transformation (Equation 24.21) does not involve data 
matrices that describe the open-loop system (Equation 24.42) and because X is assumed independent 
of 5. The reduction of conservatism by employing parameter-dependent functions X(8) is possible, to a 
certain extent, by introducing slack variables as discussed in [5]. 

Unfortunately, robust output-feedback controllers cannot be designed by following the general syn- 
thesis procedure of Section 24.6.2 for the parameter-dependent system (Equation 24.42). As the main 
reason, the convexifying transformation (Equation 24.25) involves the system data while, in addition, the 
matrix blocks in the corresponding synthesis inequalities are not affine in the parameter vector 8. Simi- 
larly, robust observers with the structure appearing in Equation 24.30 cannot be designed, just because 
the observer is defined in terms of open-loop system data. 

In contrast, for the estimation problem we have considered the convexifying transformation in Equa- 
tion 24.31 of the estimator parameters that does not involve system matrices. Therefore it is possible to 
design robust estimators in complete analogy to what was discussed for state-feedback synthesis. Specifi- 
cally, recall from Section 24.6.4 that estimator design involves a system description (Equation 24.42) with 
B(8) = 0 and E(8) = E. Then there exists an estimator (Equation 24.17) that achieves for d — ea robust 
-2-gain performance level smaller than y (according to the definition in Section 24.7.3) if there exists 
v = (X,Z, K,L, M,N) such that 


A(v, 84) + A(y, 8k) Biv, 5k) C(v, 8*) 
X(v) > 0, B(v, 8&)T -y7I D(v,8*)'} <0 forall k=1,...,N, 
C(v, 8*) D(v, 8*) = 


where X(v) is given in Equation 24.32 and 


ZA(8*) ZA(8*) ZBo(8*) 
XA(8‘) + LC(S*)+K = XA(8*) + LC(8*) | XBo(8*) + LE(8*) 
Co(3*) + ENC(S*) + EM — Co(8*) + ENC(8*) | Do(8*) + ENF(S*) 


( A(v,8*) | B(v,8*) \ 
C(v, 8) | Div,8*) ) 


are affine in the design parameters v. This set of (N + 1) LMIs can be readily implemented. If v is a solution, 
the desired robust estimator is then given by the state-space representation matrices (A;, B;, C,, D,) that 
are resolved from Equation 24.31 with nonsingular U, V satisfying Z —X = UV"Z. 


24.9 Gain-Scheduling Synthesis 


An interesting generalization of the robust controller synthesis problem treated in the previous section 
amounts to allowing the controller to be dependent on the parameter vector § = col(8;,...,8p). This 
means that the controller has online access to the time-varying parameter 8(t) through an additional 
measurement. Hence, the actual parameter value, although not known a priori, is used as extra information 
to control the system. 

The classical gain-scheduling approach towards control system design fits in this line of reasoning 
and typically amounts to associating with 8(f) a specific operating condition of the plant. A (robust) 
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controller is designed for each operating condition 5 € 8 and these are scheduled by means of switching 
or interpolation rules inferred from measurement information. This design methodology has found 
widespread applications, but the assessment of guarantees on robust stability and robust performance in 
view of time-dependent changes of 8 is usually difficult if not impossible. 

For the linear parameter-varying (LPV) system in Equation 24.42, an LPV controller is a system of the 
form 


e(t) = Ac(S(t))xc(t) + Be(S(t))y (4), 
u(t) = C,(8(t))xc(t) + D-(3(t)) y(t), 


where 8(t) satisfies the bound and rate constraints in Equation 24.36. As before, the aim will be to design 
an LPV controller (Equation 24.45) that renders the controlled system robustly stable and establishes a 
desired robust performance specification. The controlled system is given by 


E(t) = A(8(t))E(t) + B(8(t))d(t), 
e(t) = C(8(t))E(t) + D(8(t))d(t), 


where the closed-loop matrix functions are structured as in Equation 24.18. 

The synthesis techniques for the construction of LPV controllers closely resemble those for nominal 
controller design as we have investigated in Section 24.6.2. The LPV synthesis problem to achieve robust 
stability and a robust “)-gain smaller than y is then solved if one can find a controller and a smooth 
function V(8) = ¥(8)' such that for all (8, p) € 8 x p 


OX(8, 0) + A(S)1¥(8) + ¥(8).A(8)  2(8)B(8) —€(38)" 


X(8)>+0 and B(8)'X(8) -yI  D(8)' | «0, (24.46) 
C(8) D(s) =] 


(24.45) 


where 0X (8, e) is defined in Equation 24.37. Mutatis mutandis, the convexifying controller transformation 
(4X, Ac, Be, Ce; De) > (X, Y, K, L, M, N) =: v from Section 24.6 still works to arrive at a synthesis proce- 
dure for LPV output-feedback controllers. However, note that all matrices in the transformed variable v 
have become functions of 8 € or of (8, p) € 8 x ep now. In fact, finding a controller and a continuously 
differentiable ¥(-) satisfying Equation 24.46 on 8 x is equivalent to finding v(-) such that the synthesis 
inequalities 
Ziv) + A(v)'+A(v) Biv) C(v)" 
X(v) > 0, Biv)? -yI D(v)'] «0 (24.47) 
C(v) D(v) —I 

hold on all of 8 x p, where Z(v(8, p)) := diag(—OY (8, ce), OX (8, p)). This turns the LPV controller synthesis 
problem into a linear matrix-function inequality in the decision variable v(-). A description of the related 
design algorithm can be found in [1]. 

Let us consider a particular popular scenario in more detail. Suppose that the matrices in Equa- 
tion 24.42 are affine functions of § and that B, E, C, and F are actually independent of 8. Also, suppose 
that the time-varying parameters 5(t) assume their values in the polytope (Equation 24.43) without any 
constraints on their rate-of-variation. We will search for an LPV controller in which the matrix functions 
in Equation 24.45 are also affine in 8. Finally, we let V in Equation 24.46 be constant, which implies that 
OX (8, 0) = 0. 

These assumptions immediately imply that the closed-loop system matrices become affine in 8. There- 
fore, Equation 24.46 is satisfied for all 8 € 8 if and only if Equation 24.46 is satisfied for the generators 
8 = 8*,k=1,...,N, of the set 8. Hence we achieve a robust -£2-gain smaller than y for the controlled 
system if there exists Y with 


Ask) Tv + VAS‘) ABS‘) C(SK)T 
X>0 and B(Sk) TX —yI D(s‘)'] <0 fork=1,...,N. (24.48) 
C(8*) D(S*) —] 
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Under the present structural assumptions, the transformed variables (K,L, M,N) also become affine 
functions in § and Z(v) = 0. Now apply the general convexifying transformation 


(%, Ac(8*), Be(8*), Ce(8*), De(8*)) > (X,Y, Kes Lis Mies Nu) = Ve 
with k = 1,...,N as in Section 24.6.2. This transforms Equation 24.48 into 


A(vy, 88) + A(vg, 8*)  B(vg,8*) — C(vy, 8*)" 


(7 2) > 0, By, 8°)" -y7I D(y,8*)' | <0 fork=1,...,N. (24.49) 
C(vy; 8") D(vy, 8*) —I 
We end up with a system of genuine LMIs in the parameters v,...,vy that can be readily solved. In 


summary, this leads to the following LPV controller design procedure: 


¢ Verify feasibility of (or minimize y over) the synthesis inequalities (Equation 24.49) in the variables 
Vp = (X, Y, Ky, Lp, My, Nx) fork =1,...,N. 

¢ Ifa solution has been found, construct 4 as in the standard output-feedback synthesis procedure 
in Section 24.6.2. 

¢ For this fixed 1, find controller parameters (a Be ) that render Equation 24.48 satisfied for each 


k =1,...,N. These controller parameters can be obtained by solving Equation 24.25. 
« If8 € $is represented by 8 = sy a,* with a, > 0, ee a, = 1, then Equation 24.46 holds for 


N 
A-(8) B,(8) = Ack Bok 
(es a) ~ 2 a je sa 


with Y independent of 8 and, consequently, 0.V(8, p) = 0. 


The actual LPV controller is now obtained by taking time-varying convex combinations of the N 
controllers defined by the quadruples (A,4; Bek» Cok, Dex). It is given by Equation 24.45 with 


N 
A,-(8(£)) B-(8(t)) = Ack Bok 
(cise Dao) 7 ae fc ae) 


where the parameter 8(f) at time t is represented as 3(t) = yy oy (t)3* with a;(f) = 0, ee az(t) = 1. 
It achieves robust stability and a robust -gain performance smaller than y. 


24.10 Robustness Analysis and Synthesis with Multipliers 
24.10.1 Robustness Analysis with IQCs 


Uncertain systems are often described by a feedback interconnection with a stable LTI system in the 
forward path and some uncertainty in the feedback path as depicted in Figure 24.3. Mathematically, this 
amounts to 


x= Ax +Biw + Bod, 
zZ=C\x+D,w+Dj,2d interconnected with w = A(z), (24.50) 
e= C.x+ Dyw+ Dod, 

where A is Hurwitz and A denotes the uncertainty, a map that is supposed to be contained in some class 


A. We do not intend to include the theory around general systems and their abstract properties as, for 
example, developed in [6]. Instead we just assume that the interconnection is well-posed: For all A € A 
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Uncertainty 


System 


FIGURE 24.3 Uncertainty system. 


and for all initial conditions x(0) as well as arbitrary external signals d that are of finite energy on [0, T] 
for all T > 0, the interconnection has unique responses x, z, w, and e that are of finite energy on finite 
subintervals of [0, 00). 

The interconnection is said to be robustly stable if, for any A € A, arbitrary finite energy disturbance 
signals lead to responses of finite energy as well. The uncertain system has robust -7)-gain smaller than y 
if the interconnection is robustly stable and if there exists some € > 0 such that the %-gain of d > e in 
Equation 24.50 for zero initial condition is bounded by y — € for all uncertainties A € A. 

In order to computationally verify robust performance, we need to suitably capture information about 
the uncertainty set. Suppose, for example, that for all A € A and all z € & the signal w = A(z) satisfies 


il T 
/ w(t) w(t) dt < / z(t)'z(t)dt forall T>0. 
0 0 


This just means that all uncertainties have an .44-gain bounded by 1. If, in addition, we also know that all 
uncertainties are passive, we have 


i 
i w(t)'z(t)dt>0 forall T>0. 
0 


These properties of the uncertainties can be expressed as IQCs on their input and output signals with the 
help of so-called multipliers P: 


T /wit)\'. (w(t) 
[ Ey P Cy) dt>0 forall T=>0O and P€ {Pgain, Ppassive}, 


where the multipliers for the gain and passivity constraints are defined, respectively, as 


-I 0 0 L 
Pgain = ( 0 ') and Ppassive = ¢ a (24.51) 


With any set of such multipliers one can computationally verify robust performance on the basis of the 
following result. 


Theorem 24.10: 


Let P be a family of symmetric matrices P which satisfy 


T ak 
i; eau P Ce) dt>0 forall z€Al0,T],T>0 and all ACA. (24.52) 
0 z(t) z(t) 
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Then Equation 24.50 is robustly stable and has robust L2-gain performance smaller than y if there exists 


X =X! and P= (¢ ) € P that satisfy the LMIs 
“il: 
b. ioe 7 0 x|o o|o oO EB. OO 40 
A By Bo X 0/0 0/0 0 A By Bo 
0 I O 0 0/Q S|o 0 0 I  O 
X>0 and 0 (24.53 
ge EE Nos as ED 0 ols Rilo oO | be: Di De |<? -P? 
0 oOo TT 0 0 0|Q. Sp 0 Oo T 
C, Dry Dy 0 0/0 0] S, Rp C, Dy Dy 


for the performance index Pp = (2 Rp) 0 1}: 


Proof. For some sufficiently small « > 0 we can add «I to the left-hand side of Equation 24.53 while 
keeping the inequality valid. Let us then choose any trajectory of the interconnection Equation 24.50 for 
an arbitrary uncertainty. Then right-multiply Equation 24.53 with col(x(t), w(t), d(t)) and left-multiply 
it with its transpose in order to obtain 


x(t)\' (0 X)\ (x(t) wit)\ — (w(t) d(t)\'. (d(t) 
(i) Ce 0) Geo) + (25)? (25) + Cia) (a) 
+e (Ilx(é)I? + (OI? + Id (217) < 0. 


This inequality continues to hold after integration on [0, T]. As the key observation, we can exploit 
Equation 24.52 in order to conclude that one can drop the middle term in the first row without violating the 
inequality. Since the integral of the left-most term is fo (d/dt) x(t)? Xx(t) dt = x(T)'Xx(T) — x(0)'Xx(0) 
and since x(T)'Xx(T) > 0, we obtain 


T T T 
: [ lx(@)|I2 + (II? de + lee)? — y2lld(#) 12 dt < x(0)"Xx(0) — € [ Id(e) ae. 


If d is of finite energy, we infer that fo Ix)? dt and fo (122 dt remain bounded for T > oo, 
which implies that x, w, and hence also z and e (due to Equation 24.50) are of finite energy. This proves 
stability. If x(0) = 0, we conclude that the %-gain is strictly smaller than y as for Theorem 24.4. 


Clearly the very same result holds (with identical proof) for general robust quadratic performance 
(Section 24.5.4) with index 


Py = Gi Fy ) satisfying Rp > 0. 


In order to verify robust stability and performance one only needs to check feasibility of the LMI in 
Equation 24.53. This is certainly possible if P is described by computationally tractable LMI constraints 
(such as in our example in which P just consists of two elements). 

If Equation 24.52 holds for P, it persists to hold for all P in the larger convex conic hull 


N 
ci) [YonP REP W>0, v=1,...,N, ven 


v=1 


which can lead to (sometimes substantially) improved computational results. Note that cc(P) is a convex 
cone, since with any two of its elements P), P2 it also contains their positive linear combination 1, P; + 
12P2, 1, 12 > 0. Therefore, in applications, the set P in Theorem 24.10 is typically assumed to be a convex 
cone with an LMI representation. 
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24.10.2 Examples and Extensions 


Many classical results on robust performance can be obtained as special cases of Theorem 24.10. Its real 
power, however, manifests itself in the flexibility to handle mixtures of structured uncertainties beyond 
such classical cases with ease. Here is a highly nonexhaustive sample. 


24.10.2.1 Small-Gain and Passivity Theorems 


With P = {Pgain} and Py = Pgain as defined in Equation 24.51, feasibility of Equation 24.53 just means 
that the “-gain of the LTI system in Equation 24.50 is strictly smaller than one, while Equation 24.52 
translates into A € A having .2-gain smaller than one. This is simply the classical small-gain theorem. 
Similarly, for P = {Ppassive} and Pp = Ppassive, We obtain a passivity theorem which says that a passive 
interconnection in negative feedback with a passive system stays passive. 


24.10.2.2 Sector Nonlinearities 


As another variant let A(z)(t) = (z(t)) with any (Lipschitz-continuous) :IR — R whose graph is 
located in the conic sector between the lines {(x,y) € R?: y= ax} and {(x,y) € R?: y = Bx} for some 
real a < 8. For D; = 0, standard results on the existence of solutions of differential equations imply that 
Equation 24.50 is well-posed. Moreover, all these uncertainties satisfy the IQC in Equation 24.52 for all 
P € P with P being equal to 


Gee | or its conic hull {« ee =) 1TS of , (24.54) 


This holds for example, with a = 0 and B > 0 for the saturation nonlinearity 


bx for |x| <1, 
ox) = 
Bsign(x) for |x| > 1. 


For the interconnection e = z = Gw + Gd, w = A(z) with G(s) = —(12(s + 1)(s+ 2)(s + 3))|, a plot of 
the guaranteed %-gain levels over the saturation slope f is shown in Figure 24.4. It indicates a well-known 
fundamental trade-off for obtaining less conservative results for larger sets of multipliers at the expense 
of higher computational complexity. 


24.10.2.3 General Structured Uncertainties 


Let w= col(wj,...,Wq),Z = col(zi,...,2Zq), and suppose that A is diagonally structured and defined by 
wy = A\(z),v=1,...,q. If Ay satisfies an IQC with multiplier class Py, then A satisfies an IQC for all 


Energy-gain 


1 2 3 4 5 6 7 8 9 10 
Sector parameter B 


FIGURE 24.4 Results for multipliers in Equation 24.54 with a = 0: Fixed (dotted); Cone (solid). 
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P é€ P with 
diag(Q),..,Q,)  diag(Sj,...,S,) Q, § 
P:= : f . eel ie ro Py, v=l,....qF. 
{( diag(S},.S)) diag(Ri,...R,)) (si p,J S*” ” 
If the sets Py,v =1,...,q, are represented by LMIs, the same is true for P. Similarly if Py are convex 
cones for v= 1,...,q, so is the set P. This diagonal augmentation procedure allows the construction of 


multipliers for diagonally structured uncertainties of an arbitrary nature. In this fashion, we can handle 
uncertainties whose diagonals are combinations of -44-gain, passive, sector-bounded static nonlinear, or 
time-varying parametric elements as discussed below. 


24.10.2.4 Time-Varying Parametric Uncertainties 


If Ay (zy)(t) = 8)(t)z)(t) for some 8,(t) € R with |8,(t)| < 1 forall t > 0, it is trivial to check that it satisfies 
an IQC for the convex cone 


Q S 
nol( Ajraesrsed 


Diagonal augmentation leads to a set of diagonally structured multipliers for A(z)(t) = A(t)z(t) with 
A(t) = diag(31 (1, ...,84(#)D) as they are used in (frequency-by-frequency) structured singular value 
upper bound computations [20]. 


24.10.2.5 Full Block Multipliers 


If A(z)(t) = A(t)z(t) with some time-varying matrix A(t) which satisfies A(t) € co{A;,..., An}, 
a simple convexity argument reveals that Equation 24.52 holds for 


aN al 
pe [Par (3) ?(0) <° (7) 2(F) >o vate, 


Clearly this set of unstructured multipliers has an LMI description and can be readily implemented. If 
A(t) admits a diagonal structure as in the previous example, one can compare diagonally structured with 
full multipliers; the latter often lead to less conservative analysis results at a higher computational cost. 

For time-varying parametric uncertainties as considered in the latter two examples, it is remarkable to 
note that the feasibility of the LMI in Equation 24.53 does actually imply nonsingularity of I — D, A(t) and 
hence well-posedness of the interconnection (Equation 24.50); moreover one can even prove exponential 
stability of Equation 24.50. 

We have only hinted at the power of static IQCs, while the extension to allow for dynamics in the 
multipliers even further expands the wealth of the applications and their computational power [14]. 


24.10.3 Robust Synthesis 


For the purpose of synthesis, the system (Equation 24.50) is extended with a control channel as 


x= Ax+Biw+ Bod+ Bu 
z=C\x+D\w+Dpd+ Eu 
e=C)x+D.w+ Dyd+ Enu 
y=Cx+Fiw+ Fod 


interconnected with w= A(z), Ae A. (24.55) 


The goal is to design a controller (Equation 24.17) which achieves robust performance for the closed- 
loop system. With a set P of IQC multipliers satisfying Equation 24.52, we need to enforce the LMI in 
Equation 24.53 for the closed-loop system description (after having verified well-posedness). This opens 
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FIGURE 24.5 Robust estimator design. 


up the opportunity to apply the general synthesis procedure from Section 24.6.3. Convex constraints result 
if P just consists of one element P whose right-lower block is positive semidefinite, since the problem 
then boils down to one of quadratic performance synthesis as discussed in Section 24.5.4. Unfortunately, 
convexification is impossible if P consists of a whole family of multipliers. This has led to the suggestion 
of various heuristic algorithms in order to approach the solution of the bilinear matrix inequalities for 
robust performance synthesis. 

A particularly lucky case is the robust estimator synthesis problem for a configuration as depicted in 
Figure 24.5. Then Equation 24.55 specializes to B = 0, E; = 0, and Ey = —I. After having checked well- 
posedness (which does not involve the to-be-designed estimator), we can apply the convexifying estimator 
parameter transformation from Section 24.6.4. The general procedure results in the synthesis inequalities 


001 0/0 0/0 oO I 0 0 0 

65 65.00 (0! 20 6 0 i 0 0 

I 00 0}/0 0/0 0 ZA ZA ZB, ZB 
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000 0;}0 O S) Ry Cy -NC—M C,—NC)D21 — NF, D2 — NF2 


which are actually convex in X, Z and Q, S, R. Since Rp = 0 can be factorized as T}Tp with a full row-rank 
matrix T,, one can easily turn the second constraint into a genuine LMI by taking the Schur complement. 

Finally, let us stress that dualization arguments allow a similar convexification procedure for the design 
of robust state-feedback controller gains, as shown for parametric uncertainties in [16]. 


24.11 Conclusions 


We have provided a selective overview on the application of semidefinite programming techniques in 
control in order to cover the most basic but essential ideas in the analysis and synthesis of nominal and 
robust controllers. During the last decades this field has grown dramatically and the scope of applications 
of LMIs in solving complex control problems has widened considerably. Notable activities have been 
devoted to broadening the LMI stability and performance specifications and to exploring their applica- 
bility to time-varying, time-delay, fuzzy and nonlinear systems. Considerable advances have been made 
in constructing systematic relaxation schemes for handling robust LMIs. Recently emerging applications 
of convex optimization to the synthesis of distributed controllers for large-scale systems offer exiting new 
avenues for progress. 
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Appendix: Convex Sets and Convex Functions 


A set S in a linear vector space is said to be convex if 
X1,xX2 € S implies x =ax,+(1—a)x2¢€S foralla € (0,1). 


Geometrically, this states that the line segment connecting any two points of the set belongs to the set 
as well. For all a € (0, 1), the point x defined in the above expression is called a convex combination of 
x, and x2. More generally, the point x = an ax, is a convex combination of x1,...,X, € S ifar >0 
for all k and }°/_, a, = 1. Many operations preserve convexity of sets. As the most important one, 
the intersection of an arbitrary collection of convex sets is convex. Simple examples of convex sets are 
hyperplanes {x € R" | a'x = b} and half-spaces {x € R" | a'x < b}, where a € R",b ER. A polyhedron is 
the intersection of finitely many hyperplanes and half-spaces and is hence convex. A polytope is a compact 
polyhedron. It is easy to see that the set of all convex combinations of n points x1,...,x, € S is itself 
convex. For any subset S ofa linear vector space the convex hull co(S) is the set of all convex combinations 
of the elements of S. The convex hull of a finite set of points is always a polytope and, conversely, any 
polytope is the convex hull of a finite set. 
A (Hermitian-valued) function F : S > H” is convex if its domain S is convex and 


n 


n n 
(Soon) <)> oF (xx) forall x1,...,%,€S and oa, >0 with y=. 
k=1 k=1 k=1 


This is referred to as Jensen’s inequality. F is strictly convex if Jensen’s inequality holds with < in the case 
that neither of the a,’s equals one. If F is real valued, the inequalities are the same as the usual < and 
< for real numbers. Generally it is not easy to verify whether a function is convex. Twice continuously 
differentiable functions F : S — R on convex sets S with interior points are convex iff their Hessian 
satisfies 0F(x) > 0 for all x € S. If F defined on S is convex, then the sublevel sets {x € S | F(x) < H} are 
convex for any H € H’. The most important reason for considering convex functions in optimization is 
the fact that local minimal points of convex functions are actually global minimal points. Precisely, x9 € S 
is a local minimal point of F if there exists an open neighborhood N(x) of xo such that F(xo) < F(x) 
for all x € N(xo) NS. If F is convex one can conclude that F(xo) < F(x) for all x € S, showing that xo is 
a global minimal point of F. This property is of great interest in numerical optimization. Indeed, many 
efficient algorithms exist for the numerical computation of local minima of real-valued functions. If these 
are convex, such algorithms actually determine global minimal points. 
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25.1 Introduction to Optimal Control Design 


Many systems occurring naturally in fields such as biology and sociology use feedback control to achieve 
homeostasis, or equilibrium conducive to existence. Because the bounds within which life can continue 
are small (e.g., temperature changes of a few degrees can eliminate populations) and the resources available 
are limited, it is remarkable yet not expected that most of these feedback control systems have evolved 
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into optimal systems where performance objectives are achieved efficiently with a minimum of control 
effort. Since naturally occurring systems are optimal, it makes sense to design man-made controllers from 
the view point of optimality. 


25.1.1 The Philosophy of Classical Control 


Classical control theory, developed in the mid-1900s, imparts a great deal of engineering insight. It was 
best developed for linear systems. Since computers were not available to solve complex design equations, 
the design algorithms are heuristic in terms of Bode plots, Nyquist plots, the root locus, and other single- 
input/single-output (SISO) graphical techniques that offer intuition and rely on the design engineer’s 
expertise. Since most design was in the frequency domain, robustness to unknown disturbances, modeling 
errors, and noise was automatically built in. 

Complex modern systems have multiple inputs and outputs. Examples include aircraft, satellites, and 
automobile engines, which, though nonlinear, can often be linearized about a desired operating point or 
trajectory. In such applications, classical design relies on successive loop closures based on one-loop-at-a- 
time SISO design. Unfortunately, using this approach, neither stability nor robustness of the overall system 
can be guaranteed, since one loop closure can destroy what has been gained in the design of previous 
loops. 


25.1.2 The Philosophy of Optimal Control Design 


About 1960, modern optimal control theory began developing for complex multivariable systems. It 
developed coincidentally with the space age (Sputnik was launched in 1957), the computer age, and the 
age of robotics. Optimal control is a branch of modern control theory that deals with designing controls 
for dynamical systems by minimizing a performance index that depends on the system variables. The 
performance index might include, for instance, a measure of operating error, a measure of control “effort,” 
or any other characteristic important to the user of the control system. Under some mild assumptions, 
making the performance index small also guarantees that the system variables will be small, thus insuring 
closed-loop stability. 

Classical design is concerned with directly selecting the feedback gains K in the inner real-time control 
loops. On the other hand, modern control design offers standard algorithms for implementing an outer 
design loop that automatically selects the inner loop feedback gains in such a fashion that closed-loop 
stability and performance for MIMO systems is guaranteed. In contrast to classical one-loop-at-a-time 
design, in modern control, all of the feedback loops are closed simultaneously by computing the feedback 
gains by solving standard matrix design equations. Special purpose software [9,10] is commercially avail- 
able to solve these equations, so that control design for complex systems is straightforward with a personal 
computer (PC). 

In this chapter, optimal control design is discussed for deterministic systems with a well-known math- 
ematical model. The major emphasis is on linear systems in the state-space form. It will be assumed that 
full state-variable feedback is available; in the event that only partial information is available on the system 
states, the chapter on “Output Feedback” or the references should be consulted as well. The discussion 
in this chapter will center around continuous-time systems, with a follow-up discussion on discrete-time 
(DT) systems. Several design techniques will be covered, including the regulator problem, the tracker 
problem, minimum-time control, and polynomial design. Robustness of the linear quadratic regulator 
(LQR) will be discussed using multivariable frequency-domain design techniques, which allow one to 
draw close connections with classical control theory. This is a condensed version of presentations avail- 
able in [7,8,11] where derivations and computer software appear. Important key foundation references 
are [1-6]. 

There are two basic approaches to optimal control. The Calculus of Variations leads to a formulation of 
the optimal control solution in terms of partial derivatives of the Hamiltonian, which is an energy-based 
construct that captures the prescribed optimality criteria in terms of motion along the system trajectories. 
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This approach leads to optimal control solutions in terms of the state equation, costate equation, and 
stationarity condition. Bellman’s Optimality Principle, on the other hand, leads to Dynamic Programming 
(DP) solutions, which develop backwards in time. From this, one obtains minimizations with respect to 
the Hamiltonian, which leads to the Hamilton-Jacobi-Bellman (HJB) equations. First we present the 
Calculus of Variations results, then the DP results. 


25.2 Optimal Control of Continuous-Time Systems 


In this section nonlinear control design will be covered for general nonlinear systems. Then, the LQR will 
be developed for linear systems. The LQR is a cornerstone of modern control theory design. 


25.2.1 The General Continuous-Time Optimal Control Problem 


A state-variable model for a nonlinear time-varying dynamical system is given by Equation 25.1 in 
Table 25.1, where x(t) € R” is the vector of internal states and u(t) € R”™ is the vector of control inputs. 
This is the plant to be controlled. A broad range of performance objectives may be achieved by selecting 
the control u(t) to minimize a performance index (PI) or a cost given by Equation 25.2 with fo the initial 
time and T the final time of interest. The final-state weighting function (x(T), T) and weighting function 
L(x, u, t) are selected depending on the performance objectives. 

The optimal control problem is to determine a control input u(t) for the system that minimizes the PI 
and also insures that the final state constraint (Equation 25.3) is satisfied for a given function y € R?. 
The roles of the final weighting function ¢ and the final constraint iy should not be confused. The former 
is a function one would like to minimize, such as the final energy x! (T)S(T)x(T), with S(T) a specified 
weighting matrix. On the other hand, y(x(T), T) must be exactly equal to zero. A sample problem might 
be to find the control input u(t) that drives a satellite, with dynamics described by Equation 25.1, from 
a given initial position x(t) to an specified orbit, described by Equation 25.3, while minimizing the 
expended energy, as described by the PI (Equation 25.2). 


25.2.1.1 Solution of the Nonlinear Optimal Control Problem 


Using the Calculus of Variations approach to solve the optimal control problem, Lagrange multipliers 
are used to adjoin the constraints (Equations 25.1 and 25.3) to the performance index (Equation 25.2). 
Since the system Equation 25.1 is an equality constraint which must hold at each time, an associated 
multiplier X(t) € R” is required that is a function of time. Thus, the Hamiltonian function is defined as 
Equation 25.4. Using the theory of Lagrange multipliers and the calculus of variations, the solution to the 
optimal control problem given in Table 25.1 is determined. It is assumed that the initial time fo and the 
initial state x(f)) are both known and fixed. In the boundary conditions, partial derivatives are denoted 
by subscripts (e.g., yx represents oY), 

The equations in the table may be used as design equations for determining the control u(t) that 
minimizes the PI. They are necessary conditions for the solution of the nonlinear optimal control problem. 
Any control u(t) that results in a minimum value of the PI, when it is applied to the system, must satisfy 
the equations given there. Conditions under which these equations are sufficient as well are addressed 
in [3,7]. 

The structure of the equations is worth discussing. According to the table, the Lagrange multiplier X(t) 
is a dynamical variable that satisfies its own dynamical Equation 25.6; it is called the costate. The optimal 
control u(t) is then generally determined in terms of x(t) and X(t) by using the stationarity condition 
(Equation 25.7) (so named because this is the condition that guarantees a minimum or stationary point 
with respect to changes in u(f)). The value of X(t) is usually of no concern ultimately, but it is an 
intermediate variable which must be determined to solve for the optimal control u(t), that minimizes 
the PI J(to) while insuring that constraints (Equations 25.1 and 25.3) are satisfied. The appearance of 
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TABLE 25.1 Continuous Nonlinear Optimal Controller 


System model: 


x=f(x,u,t), t>to, to fixed. (25.1) 
Performance index: _ 
(ty) = o(x(T), T) + / L(x, ut) dt. (25.2) 
to 
Final state constraint: 
W(x(T), T) = 0. (25.3) 
Optimal Controller: 
Hamiltonian: 
H(x, u,t) = L(x, ut) +r" f(x, u, 6). (25.4) 
State equation: 
OH 
pA ee pS 25.5 
OD ay f, t>to (25.5) 


Costate equation: 


on _ oft. | aL 


IN = bo, t<T. 25.6 
Ox Ox Ox iz 2) 
Stationarity condition: 
dH aL aft 
== 24+. 25.7 
Ou Ou = Ou ae, 
Boundary conditions: 
x(to) given, initial condition, (25.8) 


(bx + Wlv—)! |p dx(T) + (b+ v+H) |p dT =0, final condition. (25.9) 


intermediate variables, that are required to solve for the variables of interest, is typical of optimal control 
design. 

The dynamical state and costate equations, along with the control specified by the stationarity condi- 
tion, are called the Hamiltonian system. These equations may be used to derive Lagrange’s and Hamilton’s 
equations of motion in physics (see Example 25.1). The costate equation and stationarity condition are 
called Euler’s equations. In the time-invariant case, f and L are not explicit functions of t, so that neither 
is H. In this situation 


H=0. (25.10) 


Thus for time-invariant systems and cost functions, the Hamiltonian is a constant on the optimal trajec- 
tory. This is a general statement of the principle of conservation of energy. 


25.2.1.2 Two-Point Boundary-Value Problems 


The solution for the optimal control u(t) in Table 25.1 depends on solving two coupled differential 
equations, the state Equation 25.5 and the costate Equation 25.6, each of which is of order n. These 
two dynamical equations comprise the Hamiltonian system once the stationarity condition has been 
used to eliminate u(t). The costate equation develops backward in time (by defining a backward time 
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variable t = T — t, dt = —dt), with the final condition \(T) determined by Equation 25.9. The boundary 
conditions consist of the initial conditions on the state 


nconditions: x(to) given (25.11) 

and the final conditions on the costate 
pconditions: wW(x(T),T)=0 (25.12) 
n—p conditions: (d+ wry - Wi |r dx(T)=0, (25.13) 


where it has been assumed for simplicity that the final time T is specified and hence fixed, so that dT = 0 
in condition (Equation 25.9). 

Since n boundary conditions are specified at the initial time to and n conditions are specified at the final 
time T, this is a two-point boundary-value problem. There are many methods available for solving such 
problems, including the shooting point method and the unit solution method; good software is available 
for this purpose. The solution of the optimal control problem for nonlinear systems is often difficult, 
though, for some nonlinear plants, the design equations can be explicitly solved for the optimal control 
u(t), yielding a great deal of insight. This includes the Thrust Angle Programming and Intercept and 
Rendezvous problems. In the special case that the plant is linear and the PI is quadratic, a solution is 
available, given subsequently. 


Example 25.1: Hamilton’s Principle 
Both Lagrange’s and Hamilton’s equations of motion may be derived from Table 25.1. 


LAGRANGE’S EQUATIONS OF MOTION 


Define the generalized coordinate state vector q and the “control input” as the generalized veloc- 
ities u = g. Define the Lagrangian L(q, u) = T(q, u) — U(q) as the difference between the kinetic and 
potential energies. Then the “plant” (Equation 25.1) is 


g=u=f(q,u) 
where the function f(-) is given by the physics of the problem. 


To find the trajectories of the motion, Hamilton's principle says we may minimize the performance 
index 


7 
j= L(q, u) dt, 
0 


so that the Hamiltonian (Equation 25.4) is H=L+ nu. According to Table 25.1, fora minimum 


_ 0H al 

~ 0g aq 

aH aL, 
Ou Ou 


Combining these equations yields Lagrange’s equations of motion, 


aL_ dal _, 
dq dtaq. 
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TABLE 25.2 Continuous-Time Linear Quadratic Regulator 


System model: 


x=Ax+Bu, t>to, x(to)=xo given. (25.14) 
Performance index: 
Log lf? or T 
J(to) = a (T)S(T)x(T) + sf (x* Qx + u* Ru) dt, (25.15) 
to 


with 
S(T) >0, Q>0, R>O. 


Optimal feedback control: 


Riccati equation: 
—$=A'S+SA—SBR!B'S+Q, #<T, S(T) given. (25.16) 


Optimal feedback gain: 


K=R"'B'S. (25.17) 
Time-varying feedback: 
u=—K(t)x. (25.18) 
Optimal cost: 
I(t) = 528 Stto)xo. (25.19) 


HAMILTON’S EQUATIONS OF MOTION 


Defining the generalized momentum vector by } = —OL/0q, the equations of motion may be 
expressed in Hamilton’s form as 


. OH 
q an’ 
and 
ee 
Sap 


25.2.2 Continuous-Time Linear Quadratic Regulator 


The nonlinear optimal control design equations in Table 25.1 are not easy to solve, and there is no design 
algorithm for doing so. In this section the design of optimal controllers for linear systems with quadratic 
performance indices will be discussed, the so-called LQR problem. The LQR is a cornerstone of modern 
optimal control design consisting of explicit matrix design equations easily solved on a digital computer. 
It has a wide range of relevance, because many systems are linear to begin with, and many nonlinear 
systems may be considered linear when operating near an equilibrium point. The LQR solution is given 
as a closed-loop feedback control. 

Consider the multivariable linear system (Equation 25.14) in Table 25.2 with state x € R” and control 
input u € R™. The plant matrices may be time-varying (e.g., A(t), B(t)), though for notational convenience 
this dependence will not be shown explicitly. 

Choose the control that minimizes the quadratic PI (Equation 25.15). The control weighting R, state 
weighting Q, and final state weighting S(T) are symmetric matrices of design parameters chosen by the 
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designer depending on the control objectives. For instance, if the elements of S(T) are selected larger, then 
the control will force the final state x(T) to be smaller to keep the PI small. Weight matrices Q and S(T) 
are assumed positive-semidefinite (Q > 0, S(T) > 0). Thus Q and S(T) have nonnegative eigenvalues so 
that x! Qx and x! (T)S(T)x(T) are nonnegative for all x(t). Likewise, it will be assumed that R is positive 
definite (R > 0), that is, R has positive eigenvalues so that u’ Ru > 0 for all u(t). In this case, J is always 
bounded below by zero, so that a sensible minimization problem results. Since the squares of the states 
and control inputs occur in Equation 25.15, the PI is a form of generalized energy (consider the case when 
some of the state components are velocities, or currents and voltages) and minimizing it will keep the 
states and controls small. 

Using the equations in Table 25.1, the solution to the LQR optimal control problem may be derived, 
as given in Table 25.2. The state and costate equations are 


x= Ax+Bu (25.20) 
and 
—hK=Qx+Alrx, (25.21) 


where the negative sign indicates that the costate equation must be solved backward in time. The 
stationarity condition gives the control in terms of the costate as u(t) = —R~!B" X(t). To find the optimal 
control, the two-point boundary-value problem associated with the state and costate dynamics is ana- 
lytically solved using the sweep method [3] where it is assumed that (t) = S(t)x(t) for some unknown 
auxiliary matrix S(t). The optimal control is given in terms of this intermediate matrix as follows. First, 
it can be determined that the auxiliary matrix S(t) satisfies the bilinear matrix Riccati equation 25.16. In 
terms of the Riccati solution S(t), the optimal control is given by u(t) = —R—!Bl S(t)x(t). Thus, defining 
the optimal feedback gain by Equation 25.17, one may write the optimal control as the state-feedback 
control law (Equation 25.18). 

A block diagram of the LQR is shown in Figure 25.1. It is a feedback control system with time-varying 
feedback gains K(t), and a formal design outer loop. Even if the system (A, B) is time-invariant, the 
optimal control u(t) is a time-varying state feedback. This is why the optimal LQ controller may not be 
determined using classical frequency-domain techniques. If the system model (Equation 25.14) is not an 
exact description of the plant, the LQR still performs well if it is fairly close. In fact, it will be seen in 
Section 25.2.4.2 that the LQR has important guaranteed robustness properties. 


u(t) > B >| 1/s g 
A Online 
in A 
k(t) 
-5 = ATS + SA —SBR"BTS + Q 
K=R pts Offline 


FIGURE 25.1 Linear quadratic regulator. 
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25.2.2.1 LQR Design 


To design the optimal LQR, the design engineer first selects the design parameter weight matrices Q, R, 
S(T). Then, the Riccati equation (RE) is solved for the auxiliary matrix function S(t), which is used to 
compute K(t). The RE is solved backward in time using, as a final condition, the value of the weighting 
matrix S(T) selected for the PI. This equation may be solved off-line for S(t), and the optimal feedback gain 
K(t) computed and stored. Next, a computer simulation is generally performed to verify the closed-loop 
performance. If the performance is not suitable, new design matrices Q, R, S(T) are selected and the entire 
procedure is repeated. With commercially available software, the entire process is fast and convenient. 
Finally, during the implementation or control run, the states are measured and the feedback control 
u(t) = —K(t)x applied to the plant. 

Thus, by the sweep method, the LQR problem has been decomposed into two stages: offline compu- 
tation of the optimal gains using a backward differential equation, followed by the actual control of the 
plant using feedback. Such hierarchical control schemes, consisting of an inner linear feedback loop whose 
gain is computed by an outer quadratic design equation, are typical of modern control schemes. Optimal 
control is fundamentally a noncausal design algorithm requiring future information about the plant and 
performance objectives. 

The LQR design procedure is in stark contrast to classical control design, where the gain matrix K is 
selected directly. In modern optimal control design, some parameter matrices Q, R, and S(T) are selected 
by the engineer. Then, the feedback gain K is automatically given by matrix design equations. This has the 
significant advantages of allowing all the control loops in a multiloop system to be closed simultaneously, 
while guaranteeing closed-loop stability. 

The optimal cost of using this controller is given in terms of the initial state by Equation 25.19. The 
initial state of the plant is known. Therefore, this expression allows computation of the optimal cost 
before the control is actually applied to the plant, or even before the optimal gain K(t) is computed and 
it is simulated on a computer. If the cost is too high, the engineer can select different weighting matrices 
Q, R, and S(T) in the performance index and try another design. This preview feature is typical of optimal 
control design using state feedback. 


Example 25.2: LQR for Armature-Controlled DC Motor 


The system equations of an armature-controlled DC motor are 


i= —ai—k'o+bu 
A , (25.22) 
@® = —aw + ki 


with i(t) the armature current, w(t) the motor speed, control input u(t) the armature voltage, 1/a the 
electrical time constant, 1 /a the mechanical time constant, and the remaining variables other motor 
parameters. 

Defining the state as x = [i ol, 


/ 
fe ie mate | u = Ax + Bu. (25.23) 


—a 


It is required to determine u(t) to minimize the PI 


T 
joie is ° m+ f l=" l@ 0 eee | dt (25.24) 
2 0 Sw 2 Jo 0 dw 


with s;, sq the final state weights, qi, qq the (intermediate) state weights, and r the control weight. These are 
design parameters that may be adjusted or tuned using computer simulations to yield suitable closed-loop 
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behavior, as will be seen in this example. The minimization of J corresponds to the regulation objective 
of driving the motor to a speed of zero from any initial speed, while keeping the control energy small. 
If the model represents a linearization of a nonlinear motor about a set point, the control will regulate 
the motor speed to that set point. 

Since the RE solution S(t) is symmetric, one may assume that 


s- E | (25.25) 


where the scalars s;(t) are to be determined. Substituting A, B from the state equation and S(T), Q, R from 
the PI in the RE in Table 25.2 yields the three nonlinear scalar coupled differential equations 


—s, = —2as, + 2ks2 — Bs? + dis 
—s) = —(a+a)s — k's; + ks3 — Bs152, (25.26) 


$3 = —2as3 — 2k's. — Bs} + qu» 


where B = b?/r. 
Writing the feedback gain as K(t) = [kj kj], the table shows that K = R7!B'S, so that 


b b 
ki =-S], ko = —S2. (25.27) 
r r 


Then, the optimal control is given by the time-varying feedback u = —kji — knw. 

Although the equations for s;(t) are difficult to solve analytically, it is easy to use computer soft- 
ware to solve the RE. Using such software, the optimal state trajectories and control voltage are plot- 
ted for r= 1 and several values of q = qj = qw. The results are displayed in Figure 25.2. Note that 
the states go to zero more quickly as q increases, while the controls become larger. Final weights of 
Sj = Sw = 0 were used. Based on the simulation results, suitable values for the PI weights can be selected. 
Then, the associated K(t) may be stored in memory and applied to the actual motor during the control 
implementation run. 


(a) 60- (b) 
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FIGURE 25.2 Results of DC motor simulation. (a) Motor speed. (b) Optimal control voltage. 
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25.2.3 Steady-State and Suboptimal Control 


Even for time-invariant plants, the optimal LQ control is a time-varying state-variable feedback. Such 
feedbacks are inconvenient to implement, because they require the storage in computer memory of time- 
varying gains. An alternative control scheme will now be given in which the time-varying optimal gain 
K(t) is replaced by its constant steady-state (e.g., f > oo) value. In most practical applications, this use 
of the steady-state feedback gain is adequate. 

The results of this section are important. Showing how to find multi-loop feedback gains for multi- 
input systems that are guaranteed to stabilize the closed-loop system. The gains are determined simply 
by solving a matrix design equation using computer routines available in standard software packages 
such as MATLAB® and MATRIXx™. This goes far beyond what can be achieved with classical design 
techniques, which revolve around one-loop-at-a-time procedures offering no stability guarantees. 


25.2.3.1 Steady-State Control—Guaranteed Stability of the LQR 


Suppose the plant to be controlled has the linear description 
x= Ax+ Bu, (25.28) 


with x € R” and control input u € R™. For this section it will be necessary to assume that the plant is 
time-invariant. 
Now, the control should be selected to minimize the quadratic PI 


J(to) = ; i (x" Qx + u’ Ru) dt, (25.29) 
0 


with Q > Oand R > 0. Since the integration interval is infinite, this is called an infinite horizon performance 
index; the performance objectives are referred to an infinite control interval [0, oo). 

The control law of Table 25.2 still applies; however, because the control horizon is infinite, the RE may 
reach a steady-state solution where S = 0. In this case, the RE may be replaced by the algebraic Riccati 
equation (ARE) 

0 = ATS +SA—SBR™'B'S+Q. (25.30) 


This is a symmetric matrix quadratic equation. S(T) no longer appears in this steady-state formulation. 
The ARE can have multiple solutions. However, if certain mild assumptions on the system and PI 

matrices hold, then there is a single positive definite solution S.., namely, the limiting solution to the 

time-varying RE for any S(T). Then, the optimal infinite-horizon gain is the constant matrix given by 


Keak Bie: (25.31) 
Thus, the optimal steady-state control is the constant state-variable feedback 
u(t) = —Kgox(t). (25.32) 
Moreover, the optimal cost is given in terms of the initial state by 
J= 52 OSno¥(0) (25.33) 
Under the influence of the steady-state control the closed-loop plant has the time-invariant dynamics 
x= (A— BKgo)x = Acx. (25.34) 


The advantages of this simplified control that uses a constant feedback are clear. The next theorem 
is vital to modern control theory, and shows that the steady-state LQR is guaranteed to stabilize the 
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system, even if it has multiple inputs, as long as the plant satisfies some basic properties. The system 
(Equation 25.28) is said to be stabilizable if the control input u(t) can be selected to stabilize all the modes 
in the closed-loop. This is a weaker property than reachability, which requires that there is a u(t) that 
drives any given initial state to any desired final state. These are both controllability properties of the 
plant. The system is said to be observable through the output y = Cx if measurements of only y(t) can be 
used to reconstruct the entire initial state x(0). This is a stronger condition than detectability of (A, C), 
which says that if y(t) > 0 then x(t) > 0. These are both observability properties of the plant. 

Reachability, stabilizability, observability, and detectability are basic open-loop properties that hold for 
any well-behaved system. There are simple tests for reachability and observability. In fact, if n is equal to 
the number of states, the system is reachable if, and only if, the reachability matrix 


U=[B AB A?B ... A*™1B] (25.35) 
has rank n. This implies stabilizability as well. The system is observable if, and only if, the observability 
matrix 

C 
CA 
V= (25.36) 
C. nel 


has rank n. This implies detectability as well. Standard software packages such as MATLAB and MATRIX x 
offer routines to compute these block matrices. 
The next result is a cornerstone of LQR theory. 


Theorem 25.1: Stability of Closed-Loop System 


Let C be any square root of Q so that Q= crc. Suppose (C, A) is detectable and (A, B) is stabilizable. Then: 


1. There is a unique symmetric positive-semidefinite limiting solution Soo to the Riccati equation 25.16 
independent of the choice of S(T). Furthermore, Soo is the unique positive-definite solution of the 
ARE. 

2. The closed-loop plant A, is asymptotically stable. 


This result means that, as long as the system and PI satisfy certain basic controllability and observability 
requirements, the steady-state LQ regulator will yield gains that stabilize the system. Considering the diffi- 
culty encountered by classical control techniques in stabilizing multi-input systems, this is a remarkable 
property. Exactly as in classical control theory, the theorem predicts the closed-loop stability properties of 
the system in terms of open-loop system properties that are easily tested using matrix rank techniques. 

The detectability of (./Q, A) is needed for the stability result. This property plays out as follows. If 
the infinite integral cost (Equation 25.29) has been minimized, then it has a finite value so that the 
integrand goes to zero with time. Then x7 Qx = | JMQx | * _, 0. Then detectability of (./Q, A) guarantees 
that x(t) > 0. Detectability means in fact that all the unstable plant modes should be weighted in the PI, 
which imposes a design requirement on the engineer as he selects the weighting matrix Q. 

The closed-loop poles will depend on the selection of the design matrices Q and R; however, the poles 
will always be stable as long as the engineer selects R > 0 and Q > 0 with (./Q, A) observable. Thus, the 
elements of Q and R may be varied during an interactive computer-aided design procedure to obtain 
suitable closed-loop performance. The optimal gain K is found for given values of Q and R, and the 
closed-loop time responses are found by simulation. If these responses are unsuitable, new values for Q 
and R are selected and the design is repeated. Given good software to solve for K, this procedure is quite 
convenient. Such software is available, for instance, in MATLAB and MATRIXy. 
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25.2.3.2 Suboptimal Control—Constant Feedback Gains 


Even if the control interval [0, T] is not infinite, the engineer may decide to use the steady-state gain Koo 
instead of the optimal time-varying gain K(t) given in Table 25.2. The theorems guarantee stability of 
the closed-loop system using the steady-state LOR. On a finite interval [0, T], the constant gain Koo is 
suboptimal, but the convenience gained by not having to implement a time-varying gain can more than 
compensate for the loss of optimality. Moreover, as T becomes large, the optimal gain K(f) tends to Koo 
so that the decision to use the steady-state gain makes more and more sense. In addition to the ease of 
implementation of constant feedback gains, this suboptimal controller has other important advantages: 
(1) it guarantees stability even for complex multi-loop systems, and (2) there are efficient numerical routines 
available for the solution of the ARE (e.g., MATLAB and MATRIXx). 


Example 25.3: Inverted Pendulum 


Figure 25 3 shows a rod attached to a cart through a pivot. A force u(t) is applied to the cart through 
a motor attached to an axle. The control objective is to use u(t) to balance the pendulum upright 
while simultaneously keeping the horizontal movement p(t) of the cart small. This is known as the 
inverted pendulum problem. 

The state isx =[6 6 p pj’. A force/moment balance approach or a Lagrangian approach may be 
used to obtain the dynamics. Assuming that M = 5 kg,m = 0.5 kg, L = 1 m, we thus obtain the state 
equations by linearizing the dynamics about 6 = 6 = p= p =0: 


| 0 1 0 | 0 ] 

10.78 0 0 0 —0.2 

> — 0 a X+ 0 u=Ax+ Bu. (25.37) 
—0.98 0 0 0 0.2 


The open-loop poles are at s = 0, 0, +3.283, so that, with no control input, the rod will clearly fall over 
due to the unstable pole at s = 3.283. 


It is desired to select K in 
u= —Kx = —(ko + kg + kpp + kpp) (25.38) 


to regulate the state x to zero. For this purpose, select the PI (Equation 25.29); then, K is determined 
by using the matrix design Equations 25.30 and 25.31. Values of R= 1 and Q = diag{100, 100, 10, 10} 
were selected. The motivation for choosing this Q was to place heavy emphasis on keeping the angle 0(¢) 
small; the cart position control does not matter if the rod falls over. Using MATLAB subroutines from the 
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FIGURE 25.3 Inverted pendulum on a cart. 
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FIGURE 25.4 Inverted pendulum response. (a) Angle 6(t) and position p(t). (b) Control input u(t). 


Control System Toolbox, the optimal gain was easily found to be K = [—156.16 — 49.21 —3.16 — 8.72], 
which yields closed-loop poles at s = —0.60 + j0.45, —2.48, —4.41. 

Using MATLAB routines, the closed-loop response was simulated and plotted. The angle 6(t) and 
position p(t) in response to an initial condition offset of 6(0) = 0.1rad © 6°, p(0) = 0.1m are shown in 
Figure 25.4a. The required control force u(t) is shown in Figure 25.4b. These plots are quite interesting 


and bear discussion. Due to the initial offset of 6° in angle, a large control must be applied immediately 
to push the cart under the rod to catch it so it does not fall. Subsequent smaller control motions begin to 
move the cart slowly back to the desired horizontal position of p = 0 while balancing the rod. Thus, the 
fast closed-loop poles correspond to the rod motion. The slow complex pole pair is associated with 
the cart position. This two-time scale behavior was induced by the widely disparate weightings selected in 
the design matrix Q. 

In fact, the value for Q was selected by performing several design iterations with different Q until 
computer simulation finally showed a good time response. Such design iterations, coupled with computer 
simulation, are common in modern control and are very easy using software like MATLAB. 


25.2.3.3 Eigenstructure LQR Design 


In most computer software routines for solving for the LQR gains, the ARE solution is determined from 
the eigenstructure of the Hamiltonian matrix H in the Hamiltonian system 


x A —BR7'!B")[x x 
Ls Sela cm 
which consists of the state Equation 25.20 and the costate Equation 25.21, with u(t) replaced by 
—R7'BT X(t). 
The Hamiltonian matrix H enjoys the special property of having n stable poles and n unstable poles 
(their images in the jw axis), where n is the dimension of the state vector x. If (A, B) is stabilizable, and 


(./Q, A) is observable, so that the conditions of Theorem 25.1 hold, then the stable eigenvalues of H are 
also the poles of the optimal closed-loop system 


x= (A— BKgo)x = Acx, (25.40) 


with Koo the steady-state feedback gain. This provides an alternative proof of the stability of the optimal 
closed-loop system. 
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Select the eigenvectors of the stable eigenvalues of H, and partition them as [XJ A] ]". Let X be an 
n X n matrix whose columns are X;, and A be an n x n matrix whose columns are A;. Then the solution 
to the ARE is given in terms of the eigenstructure of H by 


Soo = AX7!, (25.41) 


and the steady-state gain is given by 
Koga RB Ak, (25.42) 


(If the eigenvalues are complex, then, in the definitions of X and A, it is necessary to use the real and 
imaginary parts of the associated vectors X; and A; instead of the complex conjugate vectors themselves.) 


25.2.4 Frequency-Domain Results and Robustness of the LQR 


It is possible to discuss the LQR from the view point of the frequency domain. Classical frequency-domain 
results are given in terms of scalar SISO systems and involve notions like the loop gain, return difference, 
sensitivity, and so on. Such ideas are extended in modern optimal design using multivariable transfer 
functions and the notions of the singular value and the multivariable Bode plot. For more details, see 
Chapter 8 or the references. An important property of any closed-loop system is robustness to uncer- 
tainties, including modeling errors, disturbances, and noise. The LQR has some important robustness 
properties that are detailed here. 


25.2.4.1 LQR Frequency-Domain Relationships 
Suppose that the plant is time-invariant and, in Figure 25.1, K is the constant optimal LQ state-feedback 
gain determined using the LQR ARE as in Section 25.2.3.1. Define the plant transfer function as G(s) = 
(s] — A)~1B. Then, the loop gain referred to the input is 

KG(s) = K(sI — A)" !B = KG(s) (25.43) 
and the closed-loop return difference is [I + K(sI — A)~!B] =14+ KG(s). 


25.2.4.1.1 Optimal Return Difference Relationship 


Two key results are the following. The optimal characteristic polynomial relationship is 
A-(s) = |I+ K(sI — A)! B|A(s) (25.44) 


where the open-loop characteristic polynomial is A(s) = |s! — A| and the closed-loop characteristic poly- 
nomial is A-(s) = |sI — (A — BK)|. The optimal return difference relationship is 


[I++ K(—sI — A)~'B]R [I + K(sI — A)~'B] = R+ B! (—sI — A)~' Q(sI — A)! B. (25.45) 


These are extremely important because, exactly as in classical control theory, they express closed-loop 
properties in terms of open-loop properties that can be computed before the optimal controller is designed. 
They allow, for instance, the development of the Chang-Letov design approach for LQR which is an 
extension of root locus design to MIMO systems. 


25.2.4.1.2 Optimal Singular Value Relationships 


Select the control weighting matrix as R= pl, with p a positive design parameter. Denoting the ith 
singular value of a matrix M as o;(M), Equation 25.45 yields the optimal singular value relationship of 
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the LQR 
1 
2 


oi[I + KG(jo)] = E + “ofl jo) (25.46) 


with 
H(s) = C(sI — A)7!B (25.47) 


and matrix C defined by Q = C'C. This is important because the right-hand side is known in terms of 
open-loop quantities before the optimal feedback gain is found by solution of the ARE, while the left-hand 
side is the closed-loop return difference. 
According to this relationship, for all w, the minimum singular value, denoted , satisfies the LQ 
optimal singular value constraint 
o[I + KG(jw)] = 1. (25.48) 


Thus, the LQ regulator always results in a decreased sensitivity. 


25.2.4.2 Guaranteed Robustness of the Linear-Quadratic Regulator 


The linear-quadratic regulator using full state feedback has many useful properties, including guaranteed 
closed-loop stability and ease of design by solving matrix design equations. It will now be shown that 
the steady-state LQR has certain guaranteed robustness properties that make it even more useful. These 
conclusions may be discovered using the multivariable Nyquist criterion, which shall be referred to the 
polar plot of the return difference I + KG(s), where the origin is the critical point. A typical polar plot 
of o[I + KG(jw)] is shown in Figure 25.5, where the optimal singular value constraint appears as the 
condition that all the singular values remain outside the unit disc. 


25.2.4.2.1 Guaranteed Stability 


The multivariable Nyquist criterion says that (as long as the open-loop system G(s) is stable) the closed- 
loop system is stable if none of the singular value plots of I+ KG(jw) encircle the origin in the figure. 
Due to the optimal singular value constraint, no encirclements are possible. This constitutes a proof of 
the guaranteed stability of the LQR discussed in Section 25.2.3.1. 


25.2.4.2.2_ Gain Margin 


Multiplying the optimal feedback K by any positive scalar gain k > 1 results in a loop gain of kKG(s), 
which has a minimum singular value plot identical to the one in Figure 25.5 except that it is scaled 
outward; that is, the w — 0 limit (i.e., the DC gain) will be larger, but the w — oo limit will still be 1. 
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FIGURE 25.5 Typical polar plot for optimal LQ return difference. 
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FIGURE 25.6 Definition of multivariable phase margin. 


Thus, the closed-loop system will still be stable. In classical terms, the LQ regulator with full state feedback 
has an infinite gain margin. 


25.2.4.2.3 Phase Margin 


For multivariable systems the phase margin may be defined as the angle marked “PM” in Figure 25.6. As 
in the classical case, it is the angle through which the polar plot of o[] + KG(jw)] must be rotated (about 
the point 1) clockwise to make the plot go through the critical point. By combining Figure 25.5 with Fig- 
ure 25.6, it can be determined that, due to the LQ singular value constraint, the plot of o[I + KG(ja)] must 
be rotated through at least 60° to make it pass through the origin. The LQR with full state feedback thus 
has a guaranteed phase margin of at least 60°. This means that a phase shift of up to 60° may be introduced 
in any of the m paths in Figure 25.1, or in all paths simultaneously, as long as the paths are not coupled 
to each other in the process. (Here, m is the number of control inputs, i.e., the number of control loops.) 

This phase margin is excessive; it is higher than that normally required in classical control system 
design. This overdesign means that, in other performance aspects, the LQ regulator may have some 
deficiencies. One of these is that, at the crossover frequency (loop gain = 1), the slope of the multivariable 
Bode plot is -20 dB/decade, a relatively slow attenuation rate. (By allowing a Q weighting matrix in the PI 
that is not positive semidefinite, it is possible to obtain better LQ designs that have higher roll-off rates at 
high frequencies.) 


25.2.4.2.4 Stability with Multiplicative Uncertainty 


It can be shown that Equation 25.48 implies that 


o[I + (KG(jw))!] = (25.49) 


NI Re 


This corresponds to the fact that the LQR with state feedback remains stable for all multiplicative uncer- 
tainties in the plant transfer function which satisfies m(w) < 5. 


25.3 The Tracker Problem 


The function of the LQ regulator is to hold the states near zero, that is, to guarantee closed-loop stability. 
Another fundamental design problem in systems engineering is to control a system so that a specified 
output follows a given nonzero reference trajectory r(t). An example is controlling an aircraft to follow a 
desired step input command (e.g., change in altitude). This is called the tracking or servodesign problem. 
For this purpose, the regulator control law must be modified. The fundamental issue here is that for 
optimal tracking some additional feedforward terms must be added to the control input besides the basic 
LQR feedback loop that gives closed-loop stability. 
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Consider the linear system given by Equation 25.51, with z(t) in Equation 25.52 a performance output 
that should track the given reference input r(t). In contrast to classical control, it is easy to include here 
the case where both z(t) and r(f) are vectors, that is, the case of multivariable tracking. The control input 
is given by 


u=—Kx+y, (25.50) 


where v(t) is a feedforward signal required for good tracking performance. The feedback gain K and 
feedforward signal v(t) are determined so as to keep the tracking error (Equation 25.53) small. 

The solution to the tracking problem is significantly more involved than the regulator problem. It is 
now discussed from three points of view. In this section, full state-variable feedback is assumed. In actual 
applications, only output feedback is allowed. In that case refer to Chapter 5.1 or to [8,11], where these 
techniques are extended. 


25.3.1 Optimal LQ Tracker 


The optimal LQ tracker can be derived along the same lines as the Hamiltonian approach in Section 25.2.2. 
The result is given in Table 25.3. The feedback structure is basically the same as the LQ regulator in 


TABLE 25.3 Optimal Continuous-Time Linear Quadratic Tracker 


System model: 
xX=Ax+Bu, t>to, x(to)=xo given. (25.51) 


Performance output and tracking error: 


2= Ax, (25.52) 
e=r-z. (25.53) 
Performance index: 
I(to) = sel (DPT) + ; / el ek u! Ru) dt, (25.54) 
0 


with 
Optimal tracking controller: 
Riccati equation: 
—~$=A'S+SA—SBR'B'S+H'QH, t<T, S(T)=H"PH. (25.55) 


Optimal feedback gain: 
KoR ‘BS. (25.56) 


Feedforward system: 
—w=(A—BK)'w+H'Qr, t<T, w(T) =H! Pr(T). (25.57) 
Feedback plus feedforward control: 


u(t) = —K(t)x(t) + RB? wit). (25.58) 
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FIGURE 25.7 Optimal LQ tracker. 


Table 25.2, but now the PI contains the tracking error, not the state, because e(t) is to be small in this 
application. The presence of e(t) in the PI has the effect of adding a feedforward term v(t) to the control 
signal. The tracker gain K and signal v(t) determined with the LQ optimal approach are given in the table. 
The feedback gain K is found in the same manner as in the LQR case. The structure of the LQ optimal 
tracker is given in Figure 25.7. 

The feedforward signal is computed using the dynamical system (Equation 25.57), which is called the 
adjoint system; like the RE, it is integrated backward in time. Thus, the RE and the adjoint system must 
be integrated off-line before the control run. In fact, the optimal LQ tracker is noncausal, because future 
values of the reference input r(t) are needed to compute w(t). The ramifications of this noncausal nature 
of the optimal tracker are illustrated in Figure 25.8, which shows the optimal tracker response for a scalar 
system using control weighting R = 1 and different values of the error weighting Q. The system output 
begins to change before the reference r(t) does, so that the system anticipates the changes in r(t). This 
anticipatory behavior is an important feature of the optimal tracker. 

The noncausal nature of the optimal LQ tracker means that it cannot be implemented in practice when 
r(t) is not predetermined. Therefore, two suboptimal strategies are now outlined that yield implementable 
tracking systems. 


0.000 


FIGURE 25.8 Anticipatory response of the optimal LQ tracker. 
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FIGURE 25.9 Tracker based on DC gain. 


25.3.2 Conversion of an LOR to an LQ Tracker 


As an alternative to the optimal tracking solution just presented, a causal tracker can be obtained as 
follows. First, the LQ regulator is designed using Table 25.2. Then, it is converted to an LQ tracker by 
adding a feedforward term. In the case where the reference signal r(f) is a constant (ie., step function) 
with magnitude ro, the tracking control with state feedback is given by 


u=—Kx+H-7'(0)ro, (25.59) 
where the closed-loop transfer function is 
H.(s) = H(sI — (A— BK))~'B (25.60) 


and H,(0) is the DC gain of the closed-loop system (A — BK). The control gain K is found using the 
LQR design equations in Table 25.2. The structure of this suboptimal tracker is shown in Figure 25.9. 
Unfortunately, if the DC gain is not well known this tracker structure does not perform well, that is, this 
tracker is not robust. 


25.3.3 A Practical Suboptimal Tracker 


This section shows how to design a suboptimal tracker that works well for practical applications and is 
robust to uncertainties and disturbances. The key is in the use of engineering design insight and common 
sense to formulate the problem. One uses a unity feedback gain outer loop, which has proven effective 
in classical control approaches. This technique also relies on converting an LQR to a tracker, but differs 
from the work in the previous section. 
25.3.3.1 Problem Formulation 
A general class of systems is described by the equations 

x= Ax+Bu-+ Er 

(25.61) 
z= Hx 


which can contain both the plant plus some desirable compensator dynamics. The control input is allowed 
to have the form 
u = —Kx — KFr, (25.62) 


which consists of state feedback plus a feedforward term of a special composition. Placing the control 
into the system yields the closed-loop system 


x = (A — BK)x + (E — BKF)r = A-x+ Ber. (25.63) 
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Matrices E and F are chosen to have a structure that is sensible from a design point of view. Specifically, it 
is very desirable to incorporate a unity-gain outer tracking loop in the controller, as shown in Example 25.4. 


25.3.3.2 Deviation System and LQR Design Step 


Assume that the reference input is a unit step of magnitude ro. Then, the steady-state system is 
0O=A.x+ Boro, 


where overbars denote steady-state values, so that the steady-state value of the state is ¥ = —A>!Bero. 
Though the reference input is assumed constant for design purposes, this is to allow good closed-loop 
rise time and overshoot qualities. Then, the designed controller works for any reference input r(t), even 
though time-varying. 

Define the deviations 


ee’ ate Ae (25.64) 
u=u-—u, @=e-e 
Then, the deviations satisfy the dynamics of the deviation system 
x= AX + Bil (25.65) 
Z= Hx (25.66) 
“= —Kx. (25.67) 
Because e = r — z, the tracking error deviation is é = —Z. To induce tracking behavior, define the perfor- 
mance index 
LE ae eit ae 
J(to) = al (x* Qk + u’ Ru) dt, (25.68) 
0 


which makes the entire deviation state, and therefore é, small. 


25.3.3.3 Tracker Design 


The tracking problem may now be solved as follows. First, solve the LQ regulator problem for the deviation 
system using Table 25.2. Then, the tracking control input is given by Equation 25.62. This tracker has a 
much different structure than the DC-gain-based tracker in Figure 25.9. The next example shows that a 
sensible choice for matrices E and F based on classical control notions gives a robust tracker with a unity 
gain outer loop. Then, a sensible choice for the PI design matrices Q and R gives good control gains and 
guaranteed stability, even for complex multiloop tracking systems. 

Note that e = e + e, where é is the steady-state value of the tracking error. Because this technique only 
guarantees that é is small, special steps must be taken to guarantee that é is also small. One way to do 
this is to include integrators in all the feedforward loops, as in the next example. As an alternative, a term 
involving e can be added to the PI (Equation 25.68). This gives more involved design equations, which 
are nevertheless still easily solved by digital computer. The details are in [8]. Finally, although the gain 
determined in this fashion is optimal for the deviation system, it is not optimal for the tracking problem in 
terms of the original dynamics (Equation 25.61). In practical applications, however, it is suitable provided 
that the design matrices are sensibly selected. 


Example 25.4: Aircraft Pitch-Rate Control System 


This example illustrates the tracker design procedure just presented. Good tracker system design 
relies on a sensible selection of the structure matrices F and F, and good feedback gains rely on a 
sensible selection of the design weighting matrices Q,R. Compensator dynamics can be accounted 
for using this procedure. Because this is an LQ-based approach, a reasonable formulation of the 
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problem should result in guaranteed closed-loop stability. This is an important feature of modern 
control design techniques, and is in complete contrast to classical techniques where stability in 
multi-loop systems can be difficult to achieve. 


AIRCRAFT AND CONTROL SYSTEM DYNAMICS 


In a pitch-rate control system, the control input is elevator actuator voltage u(t) and r is a reference 
step input corresponding to the desired pitch command. The performance output z(t) is the pitch 
rate q. To ensure zero steady-state error, an integrator is added in the feedforward channel; this 
corresponds to compensator dynamics, and is easily dealt with in this approach. The integrator 
output is «. It is assumed here that all states are available as measurements for feedback purposes; in 
practice, the output-feedback design technique in [11] will be required. 

The design is based on a short period approximation to the F-16 dynamics linearized about a 
nominal flight condition of 502 ft/s, 0 ft altitude, level flight, with the center of gravity at 0.35. The 
basic aircraft states of interest are g and angle of attack a. An additional state is introduced by the 
elevator actuator, whose deflection is Se. The states of the plant plus compensator arex = [a q 8e¢ «€]! 
and the system dynamics are described by Equation 25.61 with 


[—1.01887 0.90506 —0.00215 | 
0.82225  —1.07741 —0.17555 0 
ie 0 0 202 o|’ eae) 
LE 0 —57.2958 0 0 
lo 0 
0 0 
B=|595]) F=lo]- (25.70) 
| 0 1 
and 
H=[0 572958 0 O} (25.71) 


The factor of 57.2958 is added to convert angles from radians to degrees. The last line of the state 
equation using this A and E matrix describes the integrator, € = —57.2958q +r. 


CONTROL DESIGN 


Select the control input u(t) to yield good closed-loop response to a step input at r, which corresponds 
to a SISO tracker design problem. Since the integrator makes the system Type |, the steady-state error 
é is equal to zero and e(t) = e(t). Thus, the design method just described is appropriate. 

The control input is 


u=—kx =— [ka kg ks, kj |x = kyo kqq ks, 8e kye. (25.72) 


Therefore, referring to Equation 25.62 it is evident that F = 0; however, including the integrator 
output as a state variable in the dynamics (1) adds the feedforward path required for tracking 
behavior, that is, element k; of the feedback matrix K is actually a “feedforward” gain. 

To determine the gain matrix K, select the PI (Equation 25.68), and try weighting matrices 
R=1,Q=diag{1,10,1,1}. Now use the LOR routine from the MATLAB Control Systems Toolbox 
to determine the optimal gain K = [—0.046 -— 1.072 0 3.381]. Using MATLAB routines, the corre- 
sponding closed-loop poles are s = —8.67 + j9.72, —9.85, —4.07, —1.04. The resulting step response 
is shown in Figure 25.10, which displays good performance. 
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Output 


FIGURE 25.10 Pitch-rate step response. 


25.4 Minimum-Time and Constrained-Input Design 


An important class of control problems is concerned with achieving the performance objectives in 
minimum time. A suitable performance index for these problems is 


T 
i ldt=T—t. (25.73) 
t 


0 


Several sorts of minimum-time problems are now discussed. 


25.4.1 Nonlinear Minimum-Time Problems 
Suppose the objective is to drive the system 
x =f (x, u) (25.74) 


from a given initial state x(to) € 7” to a specified final state x(T) in minimum time. Then, from Table 25.1 
the Hamiltonian is 
H=1+42'f (25.75) 


and the Euler equations are the costate equation 


. oft 
= a : (25.76) 
x 
plus the stationarity condition 
aft 
0=—-) (25.77) 
u 


Since the final state is fixed (so that dx(T) = 0) but the final time is free, the final condition in Table 25.1 


says that 
0=H(T)=1421(T)f[x(T), u(T)]. (25.78) 


If f (x, u) is not an explicit function of time, then according to the conservation principle (Equation 25.10), 
H(t) is zero for all time. 

The stationarity condition (Equation 25.77) may often be used to solve for u(t) in terms of X(t). Then, 
u(t) may be eliminated in the state and costate equations to obtain the Hamiltonian system. To solve this, 
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we require n initial conditions (x(to) given) and n final conditions (x(T) specified). However, the final 
time T is now unknown. The function of Equation 25.78 is to provide one more equation so that T can 
be solved for. Several nonlinear design problems can be explicitly solved, yielding great insight into the 
minimum-time control structure. Examples include Zermelo’s Problem and the Brachistochrone Problem. 


25.4.2 Linear Quadratic Minimum-Time Design 


The general solution procedure given in the previous section for the nonlinear minimum-time problem 
is difficult to apply. Moreover, a reasonable solution may not exist. A general class of practical problems 
is covered by the case where it is required to find an optimal control for the linear system 


x= Ax+Bu (25.79) 
that minimizes the performance index 
lr Life T "4 
J= 5% (T)Srx(T) + 5 (1+x° Qx+u’ Ru) dt (25.80) 
to 


with Sr > 0,Q > 0,R > 0, and the final time T free. There is no constraint on the final state; thus, the 
control objective is to make the final state sufficiently small. Due to the term s(T — to) arising from the 
integral, this must be accomplished in a short time period. This is a general sort of PI that allows for 
a trade-off between the minimum-time objective and a desire to keep the states and the controls small. 
Thus, if the engineer selects smaller Q and R, the term s(T — to) in the PI dominates, and the control tries 
to make the transit time smaller. This is called the LQ minimum-time problem. 

The solution for this problem is the same as in Table 25.2. The control is a linear time-varying state 
feedback given by 


Ha SRD: (25.81) 
with optimal gain 
K=R''B'S (25.82) 


and S(t) the solution determined by integrating the RE backward from time T. Unfortunately, there is a 
problem in that the final time T is unknown. 
To determine the value of T that minimizes the PI, an extra condition is needed, given by Equation 25.78, 
which yields 
x! (to)Sx(to) = 1, (25.83) 


with x(to) the specified initial condition of the plant. The solution procedure for the LQ minimum-time 
problem is to integrate the RE 


§=A™S+SA+Q-—SBR ‘B'S (25.84) 


backward from some time t using S(t) = Sr as the final condition. At each time ¢, the left-hand side of 
Equation 25.83 is computed using the known initial state and S(t). Then, the minimum interval (T — to) 
is equal to (t — t) where ¢ is the time for which Equation 25.83 first holds. This specifies the minimum 
final time T, and then allows the computation of the optimal feedback gain K(f) on the interval [to, T]. 

The Riccati derivative S is used to determine the optimal time interval, while S is used to determine the 
optimal feedback gain K(t). 

More details on this control scheme may be found in [12]. It is important to note that condition 
(Equation 25.83) may never hold. Then, the optimal solution is T — to = 0, that is, the PI is minimized by 
using no control. Roughly speaking, if x(to) and/or Q and S(T) are selected large enough, then it makes 
sense to apply a nonzero control u(t) to make x(t) decrease. On the other hand, if Q and S(T) are selected 
too small for the given initial state x(to), then it is not worthwhile to apply any control to decrease x(t), 
because a nonzero control and a nonzero time interval will increase the PI. 
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25.4.3 Constrained-Input Design and Bang-Bang Control 


Up to this point minimum-time control has been presented based on the conditions of Table 25.1, which 
were derived using the calculus of variations. Under some smoothness assumptions on f(x, u,t) and 
L(x, u, t), the resulting controls are also smooth. Here, a fundamentally different sort of control strategy 
will be presented. 
If the linear system 
x= Ax+ Bu (25.85) 


with x € R", u € R”™ is prescribed, there are problems with using the pure minimum-time PI, 


T 
J(to) = | 1 dt, (25.86) 
to 


where T is free. The way to minimize the time is to use infinite control energy! Since this optimal strategy 
is not acceptable, it is necessary to find a way to reformulate the minimum-time problem for linear 
systems. 

Therefore, the control input now must satisfy the magnitude constraint 


|u(t)|| <1 (25.87) 


for all t € [to, T]. This constraint means that each component of the m-vector u(t) must be no greater 
than 1. Thus, the control is constrained to an admissible region (in fact, a hypercube) of R”™. If the 
constraints on the components of u(t) have a value different from 1, then one may appropriately scale 
the corresponding columns of the B matrix to obtain the constraints in the form of Equation 25.87. A 
requirement like Equation 25.87 arises in many problems where the control magnitude is limited by 
physical considerations; for instance, the thrust of a rocket certainly has a maximum possible value, as 
has the armature voltage of a DC motor. 

Referring to Table 25.1, the optimal control problem posed here is to find a control u(t) that drives 
a given x(to) to a final state x(T) satisfying the final state constraint, minimizes the PI, and satisfies 
Equation 25.87 at all times. Intuitively, to minimize the time, the optimal control strategy appears to 
be to apply maximum effort (ie., plus or minus 1) over the entire time interval. This idea will now be 
formalized. When a control component takes on a value at the boundary of its admissible region (i-e., +1), 
it is said to be saturated. Pontryagin and coworkers have shown that, in the case of constrained control, 
Table 25.1 still applies if the stationarity condition is replaced by the more general condition, known as 
Pontryagin’s Minimum Principle, 


H(x*,u*, d*, t) < H(x*,u,d*,t), all admissible u. (25.88) 


This is an extremely powerful result which can be employed to derive the following solution to the linear 
constrained-input minimum-time problem. 
Define the signum function for scalar w as 


1, w>0 
sgn(w) = 4 indeterminate, w=0 (25.89) 
—l, w <0. 


If w is a vector, define v = sgn(w) as vj = sgn(w;) for each i, where v;, w; are the components of v and w. 
Then, in terms of the costate, the optimal control is given by 


u*(t) = —sgn[B" X(t)]. (25.90) 


This may be interpreted as follows. For each column J; of B, if rx" (£)B; is positive, we should select 
uj(t) = —1 to get the largest possible negative value of »/(£)b;u;(t). On the other hand, if X7(t)b; is 
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u*(t) 


BX) 


FIGURE 25.11 Sample switching function and associated optimal control. 


negative, we should select u(t) as its maximum admissible value of 1 to make 2! (t)bju;(t) as negative as 
possible. If 47 (t)b; is zero at a single point t in time, then u;(t) can be assigned any value at that time, 
because then 7 (t)bju;(t) is zero for all values of u(t). 

The quantity B’(t) is called the switching function. A sample switching function and the optimal 
control it determines are shown in Figure 25.11. When the switching function changes sign, the control 
switches from one of its extreme values to another. The control in the figure switches four times. The 
optimal linear minimum-time control is always saturated since it switches back and forth between its 
extreme values, so it is called bang-bang control. In some problems, a component b/ )(t) of the switching 
function B"(t) can be zero over a finite time interval. If this happens, component u;(t) of the optimal 
control is not well-defined by Equation 25.90. This is called a singular condition. If this does not occur, 
the time-optimal problem is called normal. 

The time-invariant plant (Equation 25.85) is reachable if, and only if, the reachability matrix (Equa- 
tion 25.35) has full rank n. If b; is the ith column of B € R”*”, then the plant is normal if 


U; =[b; Ab; ... A"~'b;] (25.91) 


has full rank n for each i= 1,2,...,m, that is, if the plant is reachable by each separate component u; of 
u €R"™. Normality of the plant and normality of the minimum-time control problem are equivalent. Let 
the plant be normal (and hence reachable), and suppose it is desired to drive a given x(to) to a desired 
fixed final state x(T) in minimum time with a control satisfying Equation 25.87. Then, the following 
results have been achieved for time-invariant plants by Pontryagin and coworkers: 


1. Ifthe desired final state x(T) is equal to zero, then a minimum-time control exists if the plant has 
no poles with positive real parts (i.e., no poles in the open right half plane). 
For any fixed x(T), if solution to the minimum-time problem exists, then it is unique. 

3. Finally, ifthe n plant poles are all real and if the minimum-time control exists, then each component 
u;(t) of the time-optimal control can switch at most n — 1 times. 


In both its computation and its final appearance, bang-bang control is fundamentally different from 
the smooth controls seen previously. The minimum principle leads to the expression (Equation 25.90) for 
u*(t), but it is difficult to solve explicitly for the optimal control. Instead, this condition specifies several 
different control laws, and it is necessary to select which among these is the optimal control. Thus, the 
minimum principle keeps one from having to examine all possible control laws for optimality, giving a 
small subset of potentially optimal controls to be investigated. In many cases, it is still possible to express 
u*(t) as a state-feedback control law. 


25-26 Control System Advanced Methods 


Example 25.5: Bang-Bang Control 


Any system obeying Newton’s laws for point-mass motion is described by 


ae, 
y (25.92) 
V=u, 


with y(t) the position, v(t) the velocity, and u(t) the input acceleration. The state is x = [y vy. 


Let the acceleration input u be constrained in magnitude by 
|u(t)| <1. (25.93) 


The control objective is to bring the state from any initial point (yo, vo) to the origin in the minimum 
time T. The final state must be fixed at 


i ON 
W(x(T), T) = Be =0. (25.94) 
Using Equation 25.88 the minimum-time control takes on only values of u = +1. Moreover, there is 


at most one control switching because the maximum number of switchings is n — 1 when the plant poles 
are all real. The phase plane is a coordinate system whose axes are the state variables. Phase-plane plots 
of the state trajectories of (1) for u=1 and for u = —1 are parabolas in the phase plane as shown in 
Figure 25.12. These parabolas represent minimum-time trajectories. The arrows indicate the direction 
of increasing time. For example, if the initial state (yo, vo) is as shown in Figure 25.12, then, under the 
influence of the control u = —1, the state will develop downward along the parabola, eventually passing 
through the point (y = 0, v = —2). On the other hand, if a control of u = 1 is applied, the state will move 
upward and to the right. 

It will now be argued that this figure represents a state-feedback control law, which brings any state 
to the origin in minimum time. Suppose the initial state is as shown in Figure 25.12. Then the only way 
to arrive at the origin, while satisfying the Pontryagin conditions, is to apply u = —1 to move the state 


—2.000 


FIGURE 25.12 Phase plane trajectories for u = 1 and u = —1. 
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along a parabola to the dashed curve. At this point (labeled “a”), the control is switched to u = 1 to drive 
the state into the origin. Hence, the resulting seemingly roundabout trajectory is in fact a minimum-time 
path to the origin. The dashed curve is known as the switching curve. For initial states on this curve, a 
control of u= 1 (if vo < 0) or u= —1 (if vp > 0) for the entire control interval will bring the state to 
zero. For initial states off this curve, the state must first be driven onto the switching curve, and then the 
control must be switched to its other extreme value to bring the final state to zero. The switching curve is 
described by the equation y = —ty lv]. 

Simply put, for initial states above the switching curve, the optimal control is u = —1, followed by 
u = 1, with the switching occurring when y(t) = 5v7(t). For initial states below the switching curve, 
the optimal control is u = 1, followed by u = —1, with the switching occurring when y(t) = — 5v7(t). 
Because the control at each time f is completely determined by the state (i-e., by the phase plane location), 
Figure 25.12 yields a feedback control law. This feedback law, represented graphically in the figure, can 
be stated as 


if 1 
-1 Rees 
or ify = —=v|v| andy <0 
u= tis (25.95) 
1 og 
orify=—>¥lvlandy > 0: 


which makes it clear that the minimum-time control is indeed a state-feedback. 


25.5 Optimal Control of Discrete-Time Systems 


The discussion so far has applied to continuous-time (analog) systems. The discussion of the LQR problem 
for DT systems 
Xk = Ax + Buz, (25.96) 


is identical in form, though more complicated in its details. The problem is to select the state-feedback 
matrix K in 
up = —Kxz (25.97) 


to minimize a performance index specified by the design engineer. 


25.5.1 Discrete-Time LQR 


In general the optimal DT linear quadratic regulator is a time-varying matrix gain sequence K,. However, 
the practically useful solution is the optimal steady-state feedback gain obtained by using the infinite 
horizon PI (Equation 25.99). The design equations for the DT LQR are given in Table 25.4. The DT LQR 
equations are more complicated than the continuous-time equivalents; however, commercially available 
software (e.g., MATLAB) makes this irrelevant to the control designer. In practice, DT design is as 
straightforward as continuous-time design. 

All the results discussed for continuous-time systems in Section 25.2 have their DT counterparts (see 
the references). Thus, as long as (A, B) is stabilizable and (A, ./Q) observable, the discrete LQR has 
guaranteed properties of stability and robustness. Discrete versions of the tracker design problem are also 
given in the references. 


25.5.2 Digital Control of Continuous-Time Systems 


Using the DT LQR design equations in Table 25.4, optimal digital controllers may be designed for 
continuous-time systems. In fact, standard techniques are available for determining a DT description 
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TABLE 25.4 Discrete-Time Linear Quadratic Regulator. 


System model: 


Xkp1 =Axp + Buz, xo given. (25.98) 
Performance index: 3 
J=5 Def Oe + uf Ri) (25.99) 
with 
Q>0, R>O0. 


Optimal feedback control: 


Discrete-time algebraic Riccati equation: 


0=S—A!SA+A!SB(BSB+R)!B! SA—Q. 25.100) 

Optimal feedback gain: 
K =(B’SB+R)—!B! SA. 25.101) 

Feedback control: 
Up = —Kxg. 25.102) 
Optimal cost: 
1 

J= 570 0: 25.103) 


given the continuous-time dynamics x = Ax + Bu and a specified sampling period T. Then, the table 
allows the design of digital controllers, because the feedback gain Equation 25.102 is expressed in DT, 
meaning that it can be directly programmed on a microprocessor or digital signal processor (DSP) and 
applied every T seconds to the plant. The next example shows some of the issues involved in digital 
control design, including selection of the sampling period and discretization of the plant. 


Example 25.6: Digital Inverted Pendulum Controller 


In Example 25.3 a continuous-time controller was designed for an inverted pendulum on a cart; it is 
now desired to design a digital controller. 


DISCRETE INVERTED PENDULUM DYNAMICS 


The continuous-time inverted pendulum dynamics are given in Example 25.3. Standard techniques 
for system discretization are covered in the chapter on Digital Control. The time histories and closed- 
loop poles in Example 25.3 reveal that a sampling period of T = 0.1 sec is very small compared to 
the speed of the plant response (e.g , about 1/10 of the smallest plant time constant). Therefore, this 
sampling period is selected. 

Using the MATLAB Control System Toolbox to compute the zero-order-hold/step-invariant sam- 
pled dynamics yields the system 


1.054386 0.101806 0 a Bee 

1.097473 1.054386 0 0 0.020361 

Xk+1= | 9004944 —0.000164 1 0.1}%**} 0.001001 | Yk =A%k + BUK 22108) 
0.099770 —0.004944 0 1 0.020033 


where the state is x, =[0, 4% Pr px]. 
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Time (s) Time (s) 


FIGURE 25.13 Response of inverted pendulum digital controller. (a) Rod angle 0(t) and cart position p(t). (b) Con- 
trol input u(t). 


The continuous system has poles at s = 0,0, 3 28, —3 28. The discrete system has poles at z = 1,1, 


1.3886, 0.7201 which corresponds to the sampling transformation z = e*!. 


DIGITAL CONTROLLER DESIGN 


To determine stabilizing control gains in 
Ux = —Kxy = —(kg Oh + kak + kpPk + kp), (25.105) 


we may use the DT LQR in Table 25.4. Note that this is a multiloop design problem, yet the LQR 
approach easily deals with it. Trying weighting matrices of R= 1,Q = diag{10, 10, 1,1} and using 
the discrete ARE solver in MATLAB yields the gains K =[—1.294 —10.02 3.648 16.94]/ and 
corresponding closed-loop poles at z = 0.37, 0.72, 0.82 + j.029. 


A simulation is easily performed to obtain the closed-loop response shown in Figure 25.13. It is very 
instructive to compare this with the response obtained in Example 25.3. The advantage of DT design is 
that the control input (2) may be computed every T = 0.1 s on a microprocessor and applied to the plant 
for real time control. The continuous-time feedback law needs to be applied using analog techniques or 
a very high sampling rate. 


25.6 Optimal LQ Design for Polynomial Systems 


The discussion thus far has focused on the state-space formulation. A dynamical system may be equally 
well described in transfer function or polynomial form as 


A(z! )yp = 2 @B(z |), (25.106) 


with y, the output and u;, the control input. The system delay is denoted d. This is a DT formula- 
tion with z~! denoting the unit delay. For simplicity we discuss the SISO case; these notions may be 
extended to multivariable polynomial systems using the matrix fraction descriptions of the plant. The 
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denominator polynomial 
A(z7!) = 1l4tayz7} 4 anz* +--+ ane" (25.107) 
has roots specifying the system poles, and the numerator 
Biz!) = bo + biz +--+ bz ™ (25.108) 


has roots at the system zeros. 
In contrast to the PI selected for state-space systems, which is a sum of squares, for polynomial systems, 
it is more convenient to select the square of sums PI 


np MQ 2 nR 2 
ice [Spe -» ani + bs ni (25.109) 
i=0 i=0 i=0 
The constants pj, qi, rj are weighting coefficients (design parameters) selected by the engineer and w; is a 


reference or command signal. Defining the weighting polynomials 


Piz ')=1+ pig | +---+Papz 
Q(z") = qo + qe | +++ + ange "2; (25.110) 
Rz"Sro tz te +i ygz", 


the PI may be written in the streamlined form 


Tic = (Pyk--a — Qe)? + (Rug)?. (25.111) 


This is a very general sort of PI. For instance, the tracking problem may be solved if we select P= Q= 1, 
R=71, for then 
Tk = Ved — We)? + (roug)? (25.112) 


and a delayed version of the output y; tries to follow a reference input wz. The system delay d is explicitly 
accounted for. Thus, the polynomial tracker is very easy to compute and implement. In fact, it is causal, 
in contrast to the state-space LQR tracker where a noncausal feedforward signal was needed. 
As another example, the regulator problem results if the weights are selected as P= 1, Q=0,R= 1, 
for then 
Tk = Vera)” + (roux)? (25.113) 


and the control tries to hold the output at zero without using too much energy. 
The optimal control u, that minimizes the PI is straightforward to determine. In the minimum-phase 
core (e.g., all roots of B(z~!) stable), one solves the Diophantine equation 


1=AF+z 4G (25.114) 


for the intermediate polynomials F (z~!) and G(z~"). Well-known routines are available for this. In fact, 
one may simply divide A(z~') into 1 until the remainder has a multiplier of z~4. Then the quotient is 
F(z~') and the remainder yields G(z~'). In terms of the Diophantine equation solution, the optimal 
control sequence is then given by the equation 


(Per PR) = SPE Owe, (25.115) 
0 


This is nothing but a difference equation that gives the current control u, in terms of yx, wg, and previous 
values of the control; it is easily implemented using a digital computer or microprocessor. Figure 25.14 
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FIGURE 25.14 Optimal polynomial LQ regulator drawn as a two-degrees-of-freedom regulator. 


shows the structure of the optimal LQ polynomial controller. Because it has a feedback and a feedforward 
component, it is called a two-degrees-of-freedom regulator. Such a controller can influence the closed-loop 
poles as well as zeros. Note that this controller actually requires full state feedback because the complete 
state is given by yp, Ve—15- - > Vk—no Uk—d> Uk—d—1> +» > UR—d—m- 

Some fundamental points in polynomial LQ design, as contrasted to state-space design, are (1) the PI 
is a square of sums, (2) the role of the RE in state-space design is played by the Diophantine equation in 
polynomial design, and (3) the optimal tracker problem is easy to solve and implement since it is causal. 


Example 25.7: Polynomial LQ Tracker 


It is desired for the plant 


3 1 
Vk — 2Yk 1+ Vk-2 = Uk—1 ~ 5 Uk-2 (25.116) 


to follow a given reference signal w, using a fairly smooth control signal uz. The control delay is 
d= 1.To accomplish the design, select the PI 


Ik = (Vena — Wa)? +1? (Ug — Ug_4)?. (25.117) 


This Pl is motivated by Equation 25.112, but the first difference of the control is weighted to keep 
Uz smooth, as per the specifications. The scalar r is a design parameter used to tune the closed-loop 
performance at the end of the design (e.g , for suitable damping ratio, overshoot, etc.). 


Inspecting the plant and PI, the polynomials defined in the discussion are 


A(z7!) = 1—227!40.75277,. B(z7!) =1—0.527!, 


(25.118) 
P(z')=Q(z"")=1, Rez") =r(1—z7). 
To find the required tracking controller, the Diophantine equation is easily solved (simply perform long 
division of A(z~!) into 1 to obtain the quotient F (z~!) and remainder z~!G(z7')), resulting in the 
intermediate quantities 


F(z7!) =1, 
: ' (25.119) 
G(z71) =2— 0.75271. 
According to Equation 25.115, therefore, the control is given by 
[d +77) — 271 (0.5 +r?) Jug = —(2 — 0.7527!) yp + wp. (25.120) 


The variable r is a design parameter that can be varied by the engineer as he performs computer 
simulations of the closed-loop system (1), (5). Then, based on the simulations, the best value of r is 
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selected and the resulting controller is applied to the actual plant. Selecting, for instance, r = 7 yields the 
difference equation 


1.25uz = 0.75ug_1 — 2yp + 0.75 yK_1 + Wks (25.121) 


which is easily solved for the current control input u,; in terms of u,_1, current and previous values of y;, 
and the current wx. The controller is of the form shown in Figure 25.14. 


25.7 Dynamic Programming and the HJB Equation 


The Calculus of Variations approach to optimal control design yields the state equation, costate equation, 
and stationarity condition in Table 25.1. A second approach to optimal control is through Bellman’s 
Optimality Principle. This principle is a cornerstone of optimal control theory and states: 


“An optimal policy has the property that no matter what the previous decisions (i.e. controls) 
have been, the remaining decisions must constitute an optimal policy with regard to the 
state resulting from those previous decisions.” 


We shall apply this to solve the optimal control problem, first for DT systems, then continuous-time 
systems. 


25.7.1 Dynamic Programming for DT Systems 


Consider the nonlinear DT plant 
Xho =f (XK) + g(xK) UK (25.122) 


with performance index over the time interval [k, N] given by 
N-1 
I(xx) = ew, N) + D5 Li, ui) (25.123) 
i=k 


It is desired to find a state variable feedback policy 


uz = h(xz) (25.124) 
that minimizes the PI. The optimal cost is 
N-1 
* — 1 . F 
rue qi (¥.99 + B L(xi, «) (25.125) 
1=. 
where u(k : N — 1) = {ug, Ug44,-°++ »UN—1}. The optimal control is 
N-1 
u* = arg min | d(xn,N)+ Ss L(xj, uj) (25.126) 
ii(k:N—1) cx 


These equations are very difficult to solve. 
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One can write (Equation 25.123) as 


N-1 
T(xp) = Lx, ue) + Olan, N)+ YO Lx, us) (25.127) 
i=k+1 
or 
T(x) = L(xg, ug) + Txr+1) (25.128) 


which is a difference equation equivalent to (Equation 25.123). That is, given a stabilizing policy u, = 
h(x;,), one can find its associated cost either by computing the infinite sum (Equation 25.123) or by solving 
the difference equation 25.128. The latter is far easier. 

Equation 25.128 is known as the Bellman Equation and is the basis for a host of reinforcement learning 
methods that learn the optimal cost and feedback policy online in real-time by observing data along the 
system trajectories. See the Chapter 6. 

In terms of the Bellman equation, one can write the optimal cost as 

Px) = pain (Erie ue) + Jsi41)) (25.129) 


it 
The importance of Bellman’s Optimality Principle is that one may write this is as 


J* (xq) = min (Le, ue) +" C41)) (25.130) 


with J* (x41) the optimal cost from time k+ 1 on. 

This is known as the Bellman optimality equation or the HJB equation. This equation yields the 
optimal cost at time k in terms of the optimal cost at time k+ 1. Therefore, it leads to a backwards-in- 
time solution procedure for the optimal control problem. This procedure leads to a host of methods for 
solution collected under the general name of Dynamic Programming. DP leads to tractable solutions in 
many situations, particularly if there are control constraints; See [7]. Since it proceeds backwards in time, 
DP is an off-line planning method for solving the optimal control problem. 

To solve the DT optimal control problem, one first solves the HJB, then finds the optimal control using 


u, = arg min (L(xz, ux) + J* (xe41)) (25.131) 
Uk 
which is far simpler to effect that Equation 25.126. 
Let us show what this boils down to in the DT LQR case. Consider the linear time-invariant DT system 
(Equation 25.96) with associated cost (Equation 25.99). For the LQR, it is known that the cost for any 
stabilizing state variable feedback (Equation 25.97) is quadratic in the state so that 


oo oo 
I(xk) = ; So xf Qui + uj Rui =; Yo xP (Q4+K™RK)x; = 5x Sx (25.132) 
i=k i=k 
For some matrix § = S’ > 0 to be determined. The closed-loop system is 
Xko1 = (A — BK) xp = Acx (25.133) 
For the DT LQR, Bellman’s equation 25.128 is 
Xj Sk = XK QUE + UR Rug + XK SKE (25.134) 


or 


Xp Sk = Xp Qxy + uj Rug + (xp + Bug)" S(Ax + Bug) (25.135) 
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whence the minimization (Equation 25.131) is easily performed by differentiating with respect to u;, to 
obtain 


Ruy +B! S(Ax, + Bug) = 0 (25.136) 
or 
up = —(R +B! SB)~!B! SAx, (25.137) 
so the optimal feedback gain is 
K =(R+B'SB)'B'SA. (25.138) 


Substituting this into the Bellman equation 25.135 and simplifying yields the DT HJB equation 
A™SA—S+Q-A'SB(R+B'SB)'B'SA =0. (25.139) 


This is exactly the DT ARE in Equation 25.100! 


25.7.2 Dynamic Programming for Continuous-Time Systems 


Consider the nonlinear dynamical system 
x = f(x, u) (25.140) 


with associated cost 


(oe) 


Hatt) = f LoCo u(t)) dt (25.141) 


t 
To apply Bellman’s optimality principle write this in the form 


t+T 
T(x(t)) = ; L(x(t), u(t))dt + J(x(t + T)) (25.142) 


t 


For any T > 0. This is exactly in the form of the DT Bellman equation. According to Bellman’s principle, 
the optimal value is given in terms of this construction as 


t+T 
J*(x(t)) = min / L(x(t), u(t))dt + J* (x(t + T)) (25.143) 
u(t:t+T) 
t 


where u(t: t+ T) = {u(t) : t <t <t+T}. The optimal control is 


t+T 


u*(x(t)) = arg min if L(x(t), u(t))dt + J* (x(t + T)) (25.144) 
u(t:t+T) 4 


Let J*(x(t+ T),t+T) =J*(x(t)+ Ax,t+T)and perform a Taylor series expansion of Equation 
25.143 and approximate the integral to obtain 


«\ T dh 
J*(x(t)) = min (urs row +(Z) ax+() r) (25.145) 
u(t:t+T) x 
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However, from Equation 25.140 Ax = f(x, u)T so that, manipulating the terms that do not depend on 
u(t) one has 


ay* T ay* T 
—{—] T= i LT — ,u)T 25.146 
( at ) est T) ( is ( a) oe keane 
whence, letting T — 0 one obtains 
ay* T ay* T 
—({—]) =min{ Ll = > 25.147 
(=) nio( +(Z Faw) seiner) 
This is the CT HJB equation. In terms of the Hamiltonian function (Equation 25.4) one may write it as 
oy? ay" 
if = ee Ps 25.148 
(ar) =a Fea) ae 


It is easily shown that, in the CT LQR case, this is exactly the CT RE (Equation 25.16). 
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26.1 Introduction 


The complexity and high performance requirements of present-day industrial processes place increasing 
demands on control technology. The orthodox concept of driving a large system by a central computer has 
become unattractive for either economic or reliability reasons. New emerging notions are subsystems, 
interconnections, distributed computing, parallel processing, and information constraints, to mention 
a few. In complex systems, where databases are developed around the plants with distributed sources 
of data, a need for fast control action in response to local inputs and perturbations dictates the use of 
distributed (that is, decentralized) information and control structures. 

The accumulated experience in controlling complex industrial processes suggests three basic reasons 
for using decentralized control structures: 


1. Dimensionality 
2. Information structure constraints 
3. Uncertainty 


Because the amount of computation required to analyze and control a system of large dimension grows 
faster than its size, it is beneficial to decompose the system into subsystems, and design controls for each 
subsystem independently based on the local subsystem dynamics and its interconnections. In this way, 
special structural features of a system can be used to devise feasible and efficient decentralized strategies 
for solving large control problems previously impractical to solve by “one-shot” centralized methods. 

A restriction on what and where the information is delivered in a system is a standard feature of 
interconnected systems. For example, the standard automatic generation control in power systems is 
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decentralized because of the cost of excessive information requirements imposed by a centralized control 
strategy over distant geographic areas. The structural constraints on information make the centralized 
methods for control and estimation design difficult to apply, even to systems with small dimensions. 

It is a common assumption that neither the internal nor the external nature of complex systems 
can be known precisely in deterministic or stochastic terms. The essential uncertainty resides in the 
interconnections between different parts of the system (subsystems). The local characteristics of each 
individual subsystem can be satisfactorily modeled in most practical situations. Decentralized control 
strategies are inherently robust with respect to a wide variety of structured and unstructured perturbations 
in the interconnections. The strategies can be made reliable to both interconnection and controller failures 
involving individual subsystems. 

In decentralized control design, it is customary to use a wide variety of disparate methods and tech- 
niques that originated in system and control theory. Graph-theoretic methods have been devised to 
identify the special structural features of the system, which may help us cope with dimensionality prob- 
lems and formulate a suitable decentralized control strategy. The concept of vector Lyapunov functions, 
each component of which determines the stability of a part of the system where others do not, is a powerful 
method for the stability analysis of large interconnected systems. Stochastic modeling and decentralized 
control have been used in a broad range of situations, involving LQG design, Kalman filtering, Markov 
processes, and stability analysis and design. Robustness considerations of decentralized control have been 
carried out since the early stages of its evolution, often preceding a similar development in the centralized 
control theory. Especially popular have been the adaptive decentralized schemes because of their flexibil- 
ity and ability to cope efficiently with perturbations in both the interactions and the subsystems of a large 
system. 

The objective of this chapter is to introduce the concept and methods of decentralized control. Due toa 
large number of results and techniques available, only the basic theory and practice of decentralized control 
will be reviewed. At the end of the chapter is a discussion of the larger background listing the books and 
survey papers on the subject. References related to more sophisticated treatment of decentralized control 
and the relevant applications are also discussed. 


26.2 The Decentralized Control Problem 


To introduce the decentralized control problem, consider two inverted penduli coupled by a spring as 
shown in Figure 26.1. The control objective is to keep the penduli in the upright position by apply- 
ing feedback control via the inputs u; and u2. The linearized equations of motion in the vicinity of 
0; = 0, = Oare 


ml, = mgl0, — ka?(0; — 92) + m1, 
m0. = mgl0, — ka*(02 — 01) + up. 


(26.1) 


By choosing the state vector x = (61, 61, 92, 6)? and the input vector u = (1, u2)', the state space repre- 
sentation of the system is 


0 1 0 0 
g ka? ka’ , 
S:x=|£ me me? u. (26.2) 
0 0 0 
ka’ Pale: ka? 
me £ me 


The fundamental restriction in choosing the feedback laws to control the system S is that each input uy 
and u2 can depend only on the local states x; = (01, 61)? and x2 = (02, 65)" of the corresponding penduli, 
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4; —> 


8; 


FIGURE 26.1 Inverted penduli. 


that is, uy = u1(x,) and uz = u2(x2). This restriction is called the decentralized information structure 
constraint. 
Since the system S is linear, a natural choice is the linear control laws 


uy = ki x, »5 “w= ki x, (26.3) 
where the feedback gain vectors k, = (k11, kz)" and kz = (k21,k22)" should be selected to stabilize the 


system S, that is, hold the penduli in the upright position. 
In control design, it is fruitful to recognize the structure of the system S as an interconnection 


5 0 1 0 0 O 0 0 
S:x,= xp+ uyt+e xj +e x2, 
lt [pferelZ, alately ol 
(26.4) 
Poe (isaac ene foc error famed rere commas Be 
2=1y 9 | 8 1 y 0 1 ie 0 25 
of two subsystems 
* 0 1 0 
$1341 =| + [5] 
a 0 ) 
(26.5) 


3 0 1 0 
si:i=|% s|>+ [5 | u2, 


where a= ¢/, B=1/ml?, y= a*k/m€, and e = (a/a)*. One reason is that, in designing control for 
interconnected systems, the designer has to account for essential uncertainty in the interconnections 
among the subsystems. Though models of the subsystems are commonly available with sufficient accuracy, 
the shape and size of the interconnections cannot be predicted satisfactorily either for modeling or 
operational reasons. In the example, the interconnection parameter e = a/a is the uncertain height of the 
spring which is normalized by its nominal value a. 

An equally important reason for decomposition is present when controlling large dynamic systems. In 
complex systems with many variables, most of the variables are weakly coupled, if coupled at all, and the 
behavior of the overall system is dominated by strongly connected variables. Considerable conceptual and 
numerical simplification can be gained by controlling the strongly coupled variables with decentralized 
control. 
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26.3 Plant and Feedback Structures 


Consider a linear constant system 


S:x=Ax+Bu, (26.6) 
y= Cx, 
as an interconnected system 
S: x; = Ajx; + Byuj + So (Aa + Byuj), 
jeN 
; (26.7) 
Vi = Cie + >> Cyx;, ieN, 
GEN 
which is composed of N subsystems 
Sj: xj = Aix; + Byuj, (26.8) 


yi= Cixi, iEN, 


where x;(t) € R™, ui(t) € R™, yi(t) € R“ are the state, input, and output of the subsystem §; at a fixed 
time t € R. All matrices have proper dimensions, and NV = {1,2,...,N}. At present we are interested in 
disjoint decompositions, that is, 
TT T)T 
X= (Xj ,%,---5XN) > 
u= Gi teed cain) 3 (26.9) 
Tet T)T 
Y= Y2>---2In) 


and where x(t) € R", u(t) € R™, and y(t) € R¢ are the state, input, and output of the overall system S, so 
that 


> 


R"” =R™ x R@ x--- x R", 
R” =R™ xR™ x--- xR”, (26.10) 


Ro=R" xR®x--- xR, 


A compact description of the interconnected system S is 
S: x =Apx+ Bpu+Acx+ Bou 
Pee ee a oe (26.11) 
y =Cpx+ Cex, 
where 
Ap = diag{Aj, A2,. se ,An}, 
Bp = diag{Bj, Bo,..., By}, (26.12) 
Cp = diag{Cy, C),. tay Cy}; 
and the coupling block matrices are 
Ac = (Ay), Bc = (Bij), Co = (Ci). (26.13) 
The collection of N decoupled subsystems is described by 
Sp: x=Apx-+ Bpu 
z a (26.14) 
y = Cpx, 


obtained from (Equation 26.11) by setting the coupling matrices to zero. 
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Important special classes of interconnected systems are input (Bc = 0) and output (Cc = 0) decentral- 
ized systems, where inputs and outputs are not shared among the subsystems. Input-output decentralized 
systems are described as 

S: x=Apx+ Bpu+Acx 
(26.15) 
y = Cpx, 
where both Bc and Cc are zero. This structural feature helps to a great extent when decentralized 
controllers and estimators are designed for large plants. 
A static decentralized state feedback, 
u=—Kpx, (26.16) 


is characterized by a block-diagonal gain matrix, 
Kp = diag{K,, Kz,..., Kn}, (26.17) 
which implies that each subsystem S; has its individual control law, 
uj = —Kjx;, ieN, (26.18) 


with a constant gain matrix Kj. The control law u of Equation 26.16, which is equivalent to the totality 
of subsystem control laws (Equation 26.18), obeys the decentralized information structure constraint 
requiring that each subsystem §; is controlled on the basis of its locally available state x;. The closed-loop 
system is described as 

§: k= (Ap — BpKpCp)x + Acx. (26.19) 


When dynamic output feedback is used under decentralized constraints, then controllers of the following 
type are considered: 


Cj: & = Fizit Giyis 


(26.20) 
uj = —Hjz; — Kiyi, 1€E N, 
which can be written in a compact form as a single decentralized controller defined as 
Cp: z=Fpz+Gpy, 
7 Dee (26.21) 
u=—Hpz-— Kpy, 
where 
ToT TyT tt Ty)T 
Z = (202 9-- ZN) > Y= oVa0-- Yn) > 
Hee ie (26.22) 
u= (Uj ,UZ,...,UN)’> 


are the state z € R", input y € R®, and output u € R™ of the controller Cp. By combining the system $ 
and the decentralized dynamic controller Cp, we get the composite closed-loop system as 


3 a i — BpKpCp+Ac | | 


: 26.23 
Z GpCp Fp Z Ge2)) 


S&Cp : 


26.4 Decentralized Stabilization 


The fundamental problem in decentralized control theory and practice is choosing individual subsystem 
inputs to stabilize the overall interconnected system. In the previous section, the plant structures have been 
described, where the plant, inputs and outputs are all decomposed with each local controller responsible 
for the corresponding subsystem. While this is the most common situation in practice, it is by no means 
all inclusive. It is often advantageous, and sometime necessary, to decentralize the inputs and outputs 
without decomposing the plant. This is the situation that we consider first. 
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26.4.1 Decentralized Inputs and Outputs 


Suppose that only the inputs and outputs, but not states, of system S in Equation 26.6 are partitioned as 
in Equation 26.9, and S is described as 


S:k=Axt > Buu, 
ieN (26.24) 
Vi= Cix, iEN. 


Then, the controllers C; of Equation 26.20 still operate on local measurements y; to generate local controls 
uj, but now they are collectively responsible for the whole system. In this case, 


_[%]_[4-BKpC —BHp] [x 
scp [3] =[ ae - HE (26.25) 


It is well-known that without the decentralization constraint on the controller, the closed-loop sys- 
tem of Equation 26.25 can be stabilized if, and only if, the uncontrollable or unobservable modes of 
the open-loop system S are stable; or equivalently, the set of (centralized) fixed modes of $, which is 
defined as 

Ac={ }o(A—BKC) (26.26) 
K 


is included in the open left half plane, where o(.) denotes the set of eigenvalues of the indicated matrix. 
This basic result has been extended in [34] to decentralized control of S, where it was shown that the 
closed-loop system Equation 26.25 can be made stable with suitable choice of the decentralized controllers 
C; if, and only if, the set of decentralized fixed modes 


Ap=(|o(A-BKpC)= [) o(A-) BKC) (26.27) 
Kp Ka easly ieN 


is included in the open left half plane. 
The result of [34] has been followed by extensive research on the following topics: 


¢ State space and frequency domain characterization of decentralized fixed modes 

¢ Development of various techniques for designing decentralized controllers (e.g., using static out- 
put feedback in all but one channel, distributing the control effort among channels, sequential 
stabilization, etc.) 

¢ Generalization of the concept of decentralized fixed modes to arbitrary feedback structure con- 
straints 

¢ Formulation of the concept of structurally fixed modes, and their algebraic and graph-theoretical 
characterization 


A useful and simple characterization of decentralized fixed modes was provided in [1]. For any subset 
T ={ij,...,ip} of the index set N, let To= {j1,-.->jn—p} denote the complement of Z in .\V, and define 


Cree ile (26.28) 
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Then a complex number » € C is a decentralized fixed mode of S if, and only if, 


26.29 
Cre 0 ( ) 


bay “3, I a 
for some Z CN. This result relates decentralized fixed modes to transmission zeros of the systems 
(A, Br, Czc), called the complementary subsystems. Thus, appearance of a fixed mode corresponds to a 
special pole-zero cancellation, which can not be removed by constant decentralized feedback. However, 
under mild conditions, such fixed modes can be eliminated by time-varying decentralized feedback. 

The characterization of decentralized fixed modes above prompts a generalization of the concept to 
arbitrary feedback structures. Let K = (ki) be an m x | binary matrix such that kij = 1 if, and only if, a 
feedback link from output y; to input u; is allowed. Thus K specifies a constraint on the feedback structure, 


a special case of which is decentralized feedback. In this case, permissible controllers have the structure 
Cr: Zi = Fizj t+ > guy; 
jeJi 


Yuji = —hi z; = DD ki 
ieJi 


(26.30) 


where Jj = {j: ki =}; 

Let K denote any feedback matrix conforming to the structure of K, that is, one with kj = 0 whenever 
kij = 0. Then, the set 

Ag =( \o(A-BKC) (26.31) 
K 

can conveniently be defined as the set of fixed modes with respect to the decentralized feedback structure 
constraint specified by K. Then the closed-loop system consisting of § and the constrained controller Cz 
can be stabilized if, and only if, A z is included in the open left half-plane. Finally, it remains to characterize 
Aj asin Equation 26.29. This, however, is quite automatic; consider the index setsZ C M = {1,2,...,M} 
and replace Z© by J = Ujerc Ji, where now T° refers to the complement of Z in M. 


26.4.2 Structural Analysis 


Structural analysis of large scale systems via graph-theoretic concepts and methods offers an appealing 
alternative to quantitative analysis which often faces difficulties due to high dimensionality and lack of 
exact knowledge of system parameters. Equipped with the powerful tools of graph theory, structural 
analysis provides valuable information concerning certain qualitative properties of the system under 
study by practical tests and algorithms [30]. 

One of the earliest problems of structural analysis is the graph-theoretic formulation of controllability 
[20]. Consider an uncontrollable pair (A, B). Loss of controllability is either due to a perfect matching of 
system parameters or due to an insufficient number of nonzero parameters, indicating a lack of sufficient 
linkage among system variables. In the latter case, the pair (A, B) is structurally uncontrollable in the sense 
that all pairs having the same structure as (A, B) are uncontrollable. Since the structure of (A, B) can be 
described by a directed graph (as explained below for a more general case), structural controllability can 
be checked by graph-theoretic means. Indeed, (A, B) is structurally controllable if, and only if, the system 
graph is input reachable (that is, each state variable is affected directly or indirectly by at least one input 
variable), and contains no dilations (that is, no subset of state variables exists whose number exceeds the 
total number of all state and input variables directly affecting these variables). These two conditions are 
equivalent to the spanning of the system graph by a minimal subgraph, called a cactus, which has a special 
structure. 

The idea of treating controllability in a structural framework has led to formulation and graph- 
theoretic characterization of structurally fixed modes under constrained feedback [26]. Let D = (V, €) 
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be a directed graph associated with the system S of Equation 26.6, where V =U UX U 9 isa set of vertices 
corresponding to inputs, states, and outputs of S, and € is a set of directed edges corresponding to nonzero 
parameters of the system matrices A, B, and C. To every nonzero aj, there corresponds an edge from 
vertex x; to vertex x;, to every nonzero bj, an edge from uj to x;, and to every nonzero cj, one from x; to 
yi. Given a feedback pattern K and adding to D a feedback edge from y; to uj for every ki = 1, one gets 
a digraph Dg = (V, € UEz) completely describing the structure of both the system S and the feedback 
constraint specified by K. 

Two systems are said to be structurally equivalent if they have the same system graphs. A system S is 
said to have structurally fixed modes with respect to a given K if every system structurally equivalent to 
S has fixed modes with respect to K. Having structurally fixed modes is a common property of a class of 
systems described by the same system graph; if a system has no structurally fixed modes, then either it 
has no fixed modes, or if it does, arbitrarily small perturbations of system parameters can eliminate the 
fixed modes. As a result, if a system has no structurally fixed modes with respect to K, then generically it 
can be stabilized by a constrained controller of the form defined in Equation 26.30. 

It was shown in [26] that a system S has no structurally fixed modes with respect to a feedback pattern 
K if, and only if 


1. all state vertices of Dg are covered by vertex disjoint cycles, and 
2. no strong component of Dg contains only state vertices, where a strong component is a maximal 
subgraph whose vertices are reachable from each other. 


This simple graph-theoretic criterion has been used in an algorithmic way in problems such as choosing 
a minimum number of feedback links (or, if each feedback link is associated with a cost, choosing the 
cheapest feedback pattern) that avoid structurally fixed modes. As an example, consider a system with a 
system graph as in Figure 26.2. Let the costs of setting up feedback links (dotted lines) from each output 
to each input be given by a matrix 


FIGURE 26.2 System graph. 
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It can easily be verified that any feedback pattern of the form 


elo [rf 


where « stands for either a 0 or a 1, avoids structurally fixed modes. Clearly, the feedback patterns which 
contain the least number of links and which cost the least are, respectively, 


= 1 0 = 0 1 
k=5 i or ka=|' A 


26.4.3 Decentrally Stabilizable Structures 


Consider an interconnected system 


S : «; = Ajx; + Bi(uj + os Dyxj) 
jEN 
yi=x ieN (26.32) 


which is a special case of the system $ in Equation 26.7 in that Aj = B;Dj, By =0, C; =I, and Cj = 0. 
Assuming that the decoupled subsystems described by the pairs (Aj, B;) are controllable, it is easy to verify 
that S has no decentralized fixed modes. Thus S can be stabilized using a decentralized dynamic feedback 
controller of the form (Equation 26.21). However, because the subsystem outputs are the states, there 
should be no need to use dynamic controllers. 

Choose the decentralized constant state feedbacks in Equation 26.18 to place the subsystem poles at 
—pip,ie N,1=1,2,...,ni, where —\i1 are distinct negative real numbers, and p is a parameter. Then 
a suitable change of coordinate frame transforms the closed-loop system of Equation 26.19 into the form 


§:x=(—pM +Ac)x, (26.33) 


where M = diag{M), M2,..., Mn}, with Mj = diag{\11, 12, ..., Min}, and Ac is independent of the 
parameter p. Clearly, § is stable for a sufficiently large p. 

The success of this high-gain decentralized stabilization technique results from the special structure 
of the interconnections among the subsystems. The interconnections from other subsystems affect a 
particular subsystem in the same way its local input does. This makes it possible to neutralize poten- 
tially destabilizing effects of the interconnections by a local state feedback and provide a high degree of 
stability to the decoupled subsystems. This special interconnection structure is termed the “matching 
conditions” [18]. 

Decentralized stabilizability of interconnected systems satisfying the matching conditions has moti- 
vated research in characterizing other decentrally stabilizable interconnection structures. Below, 
another such interconnection structure is described, where single-input subsystems are considered for 
convenience. 

Let the interconnected system be described as 


S: kj = Aix; + biuj t+ D> Ayaj, ie N (26.34) 
jEeN 
where, without loss of generality, the subsystem pairs (Aj, bj) are assumed to be in controllable canonical 
form. For each interconnection matrix Ajj, define an integer mj as 


ane ae) “ 
ry = max{q—P:apq #0}, Ay #0, (26.35) 
—n, Ay = 0, 


Thus, mj is the distance between the main diagonal and a line parallel to the main diagonal which 
borders all nonzero elements of Aj. 
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For an index set Z C N, let Zp denote any permutation of Z. Then, the system S$ in Equation 26.34 is 
stabilizable by decentralized constant state feedback if 


Yo (my —1) <0 (26.36) 

ieZ 

jeIp 
for all Z and all permutations Zp [14,30]. In the case of matching interconnections, mj; = nj — nj, so that 
Equation 26.36 guarantees decentralized stabilizability even when the elements of the interconnection 
matrices Aj are bounded nonlinear, time-varying functions of the state variables. Therefore, the condition 
(Equation 26.36) and, thus, the matching conditions, are indeed structural conditions. 


26.4.4 Vector Lyapunov Functions 


A general way to establish the stability of nonlinear interconnected systems is to apply the Matrosov- 
Bellman concept of vector Lyapunov functions [17]. The concept has been developed to provide an 
efficient method of checking the stability of linear interconnected systems controlled by decentralized 
feedback [30]. First, each subsystem is stabilized using local state or output feedback. Then, for each 
stable closed-loop (but decoupled) subsystem, a Lyapunov function is chosen using standard methods. 
These functions are stacked to form a vector of functions, which can then be used to form a single 
scalar Lyapunov function for the overall system. The function establishes stability if we show positivity 
of the leading principal minors of a constant aggregate matrix whose dimension equals the number of 
subsystems. 
Consider the linear interconnected system of Equation 26.7, 


S: kj = Aixi t+ Biuj+ )~ ejAgx;, i€N, (26.37) 
jEeN 


where the output y; is not included and Bj =0. We inserted the elements of ej € [0,1] of the 
N x N interconnection matrix E = (ej) to capture the presence of uncertainty in coupling between the 
subsystems 


Sj: xj = Ajx; + Bjuj, (26.38) 
as illustrated by the example of the two penduli above. 
We assume that each pair (Aj, Bj) is controllable and assign the eigenvalues —o} +jw},..- —op, ss 
Op, Boss Ody 413 er =o), to each closed-loop subsystem 
S; : Xj = (Aj — Bi Ki) x; (26.39) 


by applying decentralized feedback 


uj = —Kjx;. (26.40) 
Using a nonsingular transformation, 
xi= Tj Xi, (26.41) 
we can obtain the closed-loop subsystems as 
Si: Xi = AiXi, (26.42) 


where the matrix Aj = T;- 1(4; — B;K;)T; has the diagonal form 


. OY —o, wi, 
Aj = diag | i» “] Saks oc = »—O2pj-+1> +++ On; f- (26.43) 
1 1 ; 
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For each transformed subsystem, there exists a suitable Lyapunov function v : R” — R4 of the form 
2 eee 
vi(Xi) = (Xj) HiXi)?, (26.44) 
where Hj = J; is the solution of the Lyapunov matrix equation 
Aj,H, + HiAj = -—G; (26.45) 


for Gj = diag{o},0/,..., bi Tap Lod shh: 
To determine the stability of the overall interconnected closed-loop system 


S: j= AK + D~ eg AgX (26.46) 
jEN 


from the stability of the decoupled closed-loop subsystems $;, we consider subsystem functions v; as 
components of a vector Lyapunov function v = (v1, v2,..., vw)!, and form a candidate Lyapunov function 
V : R” > R, for the overall system S as 


Ve) = do divi), (26.47) 
ieN 


where the existence of positive numbers d; for stability of $ has yet to be established, and Ay =T; TAyT}. 
Taking the total time derivative of V(x) with respect to S, after lengthy but straightforward computa- 
tions [30], 


V(x) < —d’ Wz, (26.48) 
with d = (dj, d2,...,dy)?, z= (Ixill, [2]... .» [lav ll)7, and W = (ij) is the N x N aggregate matrix 
defined as 

4 aay b/2r ae Ss ah 
i — beh A: Ai), = 

nj = ae on (26.49) 
ip hay (A; Ai), ixj 


where oi, is the minimal value of all Ot and ys(- ) is the maximal eigenvalue of the indicated matrix. 
The elements é;; of the fundamental interconnection matrix E= (é;) are binary numbers defined as 
- 1, S; acts on S; 
a i J t 
pis 26.50 
y {0 S; does not act on §;. ( ) 


In this way, the binary matrix describes the basic interconnection structure of the system S. In the case of 


two penduli, 
= 1 1 
E= | : (26.51) 


It has been shown in [30] that stability of —W (all eigenvalues of —W have negative real parts) implies 
stability of the closed-loop system § and, hence, §. To explain this fact, we note first that w;j > 0, wi < 
0 (i Aj), which makes W an M-matrix (e.g., [30]) if, and only if, there exists a positive vector d (dj > 
0, i€ AN), so that the vector 

cl=d'W (26.52) 


is a positive vector as well. Positivity of c and d imply V(x) > 0 and V(x) < 0 and, therefore, stability 
of § by the standard Lyapunov argument. Finally, the M-matrix property of W is equivalent to stability 
of —W. 
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Several comments are in order. First, we note that the M-matrix property of W can be tested by a 
simple determinantal condition 


Wit Wi2 Wik 
War W22 ++. Waklg keN. (26.53) 
Wk We Wk 


Another important feature of the concept of vector Lyapunov functions is the robustness information 
about decentrally stabilized interconnected system S. The determinantal condition (Equation 26.53) is 
equivalent to the quasidominant diagonal property of W, 


N 
wi > dso djlivjl, iE N. (26.54) 
jx 


where the d;’s are positive numbers. From Equation 26.54, it is obvious that, if W is an M-matrix, so is W 
for any E < E, where the inequality is taken element by element; the system $ is connectively stable [30]. 
When a system is connectively stabilized by decentralized feedback, stability is robust and can tolerate 
variations in coupling among the subsystems. When the two penduli are stabilized for any given position 
a of the spring, including the entire length ¢ of the penduli, the penduli are stable for any position a < a. 
In other words, if the penduli are stabilized for the fundamental interconnection matrix E of (51), they 
are stabilized for any interconnection matrix 


E= p d (26.55) 


whenever e € [0, 1]. 

Finally, the decentrally stabilized system can tolerate nonlinearities in the interconnections among the 
subsystems. The nonlinear interconnections need not be known since only their size is required to be 
limited. Once the closed-loop system § is shown to be stable, it follows [30] that a nonlinear time-varying 
version 

Sy 2%) = (Ai — BK) x; +hi(t,%), ieN (26.56) 


of § is connectively stable, provided the conical constraints 
N 
Whit, 2) < Do eye sll¥l, teN (26.57) 
j=l 


on interconnection functions h; : R x R" > R™ hold, where the nonnegative numbers &j do not exceed 


nay (AE AY). This robustness result is useful in practice because, typically, interconnections are poorly 
known, or they are changing during operation of the controlled system. 


26.5 Optimization 


There is no general method for designing optimal decentralized controls for interconnected systems, 
even if they are linear and time invariant. For this reason, standard design practice is to optimize each 
decoupled subsystem using linear quadratic (LQ) control laws. Then, suboptimality of the interconnected 
closed-loop system, which is driven by the union of the locally optimal LQ control laws, is determined 
with respect to the sum of the quadratic costs chosen for the subsystems. The suboptimal decentralized 
control design is attractive because, under relatively mild conditions, suboptimality implies stability. 
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Furthermore, the degree of suboptimality can serve as a measure of robustness with respect to a wide 
spectrum of uncertainties residing in both the subsystems and their interactions. 
Consider again the interconnected system 


S: X= Ajxjt+ Buu +) Ayxy, ie N (26.58) 
ieN 
in the compact form 
S:x=Apx+Bpu+Acx. (26.59) 


We assume that the subsystems 


Sj 2%; = Ajxj + Byu; (26.60) 
or, equivalently, their union 
S:x=Apx+Bpu, (26.61) 


is controllable, that is, all pairs (A;, B;) are controllable. 
With Sp we associate a quadratic cost 


Ip(xo, u) = , (x? Qpx + u? Rpu) dt, (26.62) 
0 


where Qp = diag{Q), Qo,...,; Qn} isa symmetric nonnegative definite matrix, Rp = diag{R,, R2,...,Rn} 


: ; ve . ‘ ' 1/2, . ; 
is a symmetric positive definite matrix, and the pair (Ap, Q Ee ) is observable. The cost Jp can be considered 
as a sum of subsystem costs 


CO 
Jilxio, ui) = (xj Qix; + uj Ruj) dt. (26.63) 
0 


In order to satisfy the decentralized constraints on the control law, we solve the standard LQ optimal 
control problem (Sp, Jp) to get 
ud = —Kpx, (26.64) 


where Kp = diag{Kj, K2,..., Ky} is given as 
Kp = Rp'BpPp; 


and Pp = diag{P), P2,..., Px} is the unique symmetric positive definite solution of the algebraic Riccati 
equation 
ABPp + PpAp — PpBpRp BpPp + Qp = 0. (26.65) 


The control up, when applied to Sp, results in the closed-loop system 
SS : x = (Ap — BpKp) x, (26.66) 
which is optimal and produces the optimal cost 
TS (xo) = x4 Poxo. (26.67) 
The important fact about the locally optimal control uv is that it is decentralized. Each component 
uo = —K;x; (26.68) 


of uS uses only the local state x;. Generally, the proposed control strategy is not globally optimal, but we 
can proceed to determine if the cost J; (xo) corresponding to the closed-loop interconnected system 


§® -x= (Ap — BpKp + Ac) x (26.69) 
is finite. If it is, then S® is suboptimal and a positive number |1 exists such that 
T® (x0) < eS (x0) (26.70) 


for all x) € R”. The number w is called the degree of suboptimality of ud. 
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We can determine the index by first computing the performance index 
T2 (xo) = xd Hxo, (26.71) 


where 


CO 
H= i. exp(A!t)Gp exp(At) df, 
0 
Gp = Qp + PpBpRp' BPp, (26.72) 


and the closed-loop matrix is 
A = Ap —BpKp+Ac. (26.73) 


It is important to note that ud is suboptimal if, and only if, the symmetric matrix H exists. The existence 
of H is guaranteed by the stability of S, in which case we can compute H as the unique solution of the 
Lyapunov matrix equation 

A'H+HA=-Gp. (26.74) 


The degree of suboptimality, which is the largest we can obtain in this context, is given as 


ve ie 


(HP>5'). (26.75) 

Details of this development, as well as the broad scope of suboptimality, were described in [30], where 
special attention was devoted to the robustness implications of suboptimality. First, we can explicitly 
characterize suboptimality in terms the of interconnection matrix Ac. The system $® is suboptimal with 
degree if the matrix 


F(\) = ACPp + PpAc — (1— 2)(Qp + PoBpRp' BpPp) (26.76) 


is nonpositive definite. This is a sufficient condition for suboptimality, but one that implies stability if the 
pair {Ap + Ac, qi/ 4 is detectable. 

Another important aspect of nonpositivity of F(\1) is that it implies stability even if each control u® is 
replaced by a nonlinearity ;(u©), which is contained in a sector, or by a linear time-invariant dynamic 
element. Furthermore, if the subsystems are single-input systems, then each subsystem feedback loop has 


infinite gain margin, at least + cos~!(1 — 5) phase margin, and at least 501% gain reduction tolerance. 
These are the standard robustness characteristics of an optimal LQ control law, which are modified by 
the degree of suboptimality. It is interesting to note that the optimal robustness characteristics can be 
recovered by solving the inverse problem of optimal decentralized control. The matching conditions are 
one of the conditions that guarantee the solution of the problem. 

The concept of suboptimality extends to the case of overlapping subsystems, when subsystems share 
common parts, and control is required to conform with the overlapping information structure constraints. 
By expanding the underlying state space, the subsystems become disjoint and decentralized control can 
be designed for the expanded system by standard techniques. Finally, the control laws obtained are 
contracted for implementation in the original system. This expansion-contraction framework is known 
as the inclusion principle. For a comprehensive presentation of this principle, see [30]. 


26.6 Adaptive Decentralized Control 


As mentioned in the section on decentrally stabilizable structures, many large scale interconnected sys- 
tems with a good interconnection structure can be stabilized by a high-gain type decentralized control. 
How high the gain should be depends on how strong the interconnections are. If a bound on the inter- 
connections is known, then stability can be guaranteed by a fixed high-gain controller. However, if such a 
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bound is not available, then one has to use an adaptive controller which adjusts the gain to a value needed 
for overall stability. 
Consider an interconnected system consisting of single-input subsystems 


S: x(t) — Ajx;(t) + b;[u;(t) + h(t, x(t))], ieN (26.77) 


where, without loss of generality, the pairs (Aj, bj) are assumed to be in controllable canonical form, and 
the nonlinear matching interconnections h; : R x R” — R are assumed to satisfy 


|hi(t,x)| < Y> oujllxjll (26.78) 
jEN 


for some unknown constants aj = 0. Leta decentralized state feedback 
uj(t) = —p(t)k} Ri(p(t)), ie N (26.79) 


be applied to S, where Rj(p) = diag{p”'—!,..., p, 1}, with p(t) being a time-varying gain, and kT are 
such that the matrices Aj = Aj- bik} have distinct eigenvalues ij, i¢ V,1=1,2,...,n;. Let T; denote 
the modal matrices of Ai, ie., TiAiT; ' = Mj = diag{i1, di2,.--, hin, }. Then a time-varying coordinate 
transformation z;(t) = T;R;(p(t))x;(t) transforms the closed-loop system S$ into 


§: z(t) = p()Mizi(t)+gi(t.2()), ieN, (26.80) 


where, provided 0 < p(t) < 1 < p(t), 


ligi(t, z)II < >> Ballz (26.81) 
jEN 


for some unknown constants Bij = 0. From Equations 26.80 and 26.81 it follows that there exists a p* > 0 
so that § is stable for all p(f) satisfying 0 < p(t) < 1 < p* < p(t), as can be shown by the vector Lyapunov 
approach. However, the crucial point is that p* depends on the unknown bounds §j. Fortunately, the 
difficulty can be overcome by increasing p(t) adaptively until it is high enough to guarantee stability of S. 
A simple adaptation rule that serves the purpose is 


p(t) = min{1, yllx(¢)|I} (26.82) 


where y > 0 is arbitrary. Although the control law is decentralized, p(t) is adjusted based on complete 
state information. 

The same idea can also be used in constructing adaptive decentralized dynamic output feedback con- 
trollers for various classes of large scale systems with structured nonlinear, time-varying interconnections. 
A typical example is a system described by 


S : xj(t) = Ajxj(t) + bju;(t) + hy(t, x(t), 


26.83 
y(t) = cd xj(t), ieN oe, 


where 


1. The decoupled subsystems described by the triples (Aj, bj, ee ) are controllable and observable. 
2. The transfer functions G;(s) = ce (sI — A;)~ 1b; of the decoupled systems are minimum phase, have 
known relative degree qj and known high frequency gain kj = lims_, 99 s#G;(s). 
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3. The nonlinear interconnections h; :R x R” > R” are of the form h(t, x) = bifi(t, x) + gilt, y) 
where fj: R x R” > Rand g;: R x R" > R” satisfy 


Lil x0) < D> of Ix 
GEN 


& 
ligi(tsx)I < D> of yy 
jEN 


(26.84) 


for some unknown constants at, oti where x(t) = Gi Oe OI and y(t) = [yi(t), 


yi (t),... yn (t)]" are the state and the output of the overall system. 


Finally, suitable adaptive decentralized control schemes can be developed by forcing an interconnected 
system of the form (Equation 26.83) to track a decoupled stable linear reference model described as 


Su: xmi(t) = Amixmi(t) + bmiri(t), 


(26.85) 
ymilt) = cyxmilt), ie N, 


under reasonable assumptions on S and Sy. 


26.7 Discrete and Sampled-Data Systems 


Most of the results concerning the stability and stabilization of continuous-time interconnected systems 
can be carried over to the discrete case with suitable modifications. Yet, there is a distinct approach to the 
stability analysis of discrete systems, which is to translate the problem in to that of a continuous system 
for which abundant results are available. For an idea of this approach, consider a system 


Ssp :x(t+1) = (Ao+ — peAg)x(t) (26.86) 
kek 


where Ag is a stable matrix additively perturbed by p, Ax, ke K = {1,2,...,K} with p; standing for one 
of K perturbation parameters. The purpose is to find the largest region in the parameter space within 
which Sp remains stable. By choosing a Lyapunov function v(x) = x! Px, where P is the positive definite 
solution of the discrete Lyapunov equation, 


Ad PAy — P= —I, (26.87) 
it can be shown that Ssp is stable, provided I — W(p) is positive definite, where 


W(p) = )— pe(AgPAo +Ag PAK) + >> pxprAg PAL. (26.88) 
kek kleK 


Since the perturbation parameters appear nonlinearly in W(p), characterization of a stability region in 
the parameter space is not easy. However, I — W(p) is positive definite if the continuous system 


E(t) =(—I+ )— prEp&(t) (26.89) 
kek 
is stable, where ; 
0 Pl/2 A, 
sia Fez ATPAy + Al | 26.20) 


An analysis of the stability of the perturbed continuous system in Equation 26.89 provides a sufficient 
condition for the stability of the discrete system in Equation 26.86. This idea can be generalized to the 
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stability analysis of discrete interconnected systems by treating the interconnections as perturbations to 
nominal stable decoupled subsystems. 

A major difference between discrete and continuous systems is that characterizing decentrally stabiliz- 
able interconnections for discrete systems is not as easy as for continuous systems. For example, there is 
no discrete counterpart to the matching conditions. On the other hand, most existing control schemes for 
continuous systems seem applicable to sampled-data systems provided the sampling rate is sufficiently 
high. To illustrate this observation, consider the decentralized control of an interconnected system, 


S: s(t) = Aixi(t)+ blu()+ > djl ieNn, (26.91) 
jen 
using sampled-data feedback of the form 


uj(t) = —k} (t — tm) xi(tm)> tm <t < tm41s (26.92) 


where f,, are the sampling instants, and k;(t) are time-varying local feedback gains. With Ty, = tin41 — tm 
denoting the mth sampling period, it can be shown that the choice of 


ki? (t) = [8"(t)...8/(t) 8(£)] (26.93) 


or similar feedback gains having impulsive behavior, stabilize § provided T,, are sufficiently small. How 
small the sampling periods should be requires knowledge of the bounds on the interconnections. If these 
bounds are not available, then a simple centralized adaptation scheme, such as 


Tata la ye Villxi(tm—m) ||, (26.94) 
jEeN 


with y; > 0, decreases Tin to the value needed for stability. Clearly, this is a high-gain stabilization scheme 
coupled with fast sampling, owing its success to the matching structure of the interconnections [36]. 
Similar adaptive sampled-data control schemes are available for more general classes of interconnected 
systems. 


26.8 Graph-Theoretic Decompositions 


Decomposition of large scale systems and their associated problems is often desirable for computational 
reasons. In such cases, decentralization or any other structural constraints on the controllers, estimators, 
or the design process itself, is preferred rather than necessary. Depending on the particular problem in 
hand, one may be interested in obtaining lower block triangular (LBT) decompositions, input and/or 
output reachable acyclic decompositions, e-decompositions, overlapping decompositions, etc. [30]. In 
all of these decomposition schemes, the problem is to find a suitable partitioning and reordering of the 
input, state, or output variables so that the resulting decomposed system has some desirable structural 
properties. As expected, the system graph plays the key role, with graph-theory providing the tools. 


26.8.1 LBT Decompositions 


LBT decompositions are used to reorder the states of system S in Equation 26.6, so that the subsystems 
have a hierarchical interconnection pattern as 


i 
S24) = D> Ajay + Biu, ieN, 
j=l 


y= os Cixi}. (26.95) 
ieN 
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Such a decomposition corresponds to transforming the A matrix into a Lower Block-Triangular form 
by symmetric row and column permutations (hence the name LBT decomposition). In terms of system 
graph, LBT decomposition is the almost trivial problem of identifying the strong components of the 
truncated digraph D, = (4, Ex), where €, C € contains only the edges connecting state vertices. 

LBT decompositions offer computational simplification in the standard state feedback or observer 
design problems. For example, the problem of designing a state feedback 


u=—Kx=—)> Kix (26.96) 
ieN 
for arbitrary pole placement, can be reduced to computation of the individual blocks K; of K ina recursive 
scheme involving the subsystems only. 


26.8.2 Acyclic IO Reachable Decompositions 


In acyclic input-output (IO) reachable decompositions, the purpose is to decompose S into the form 


i i 
j=l j=l 


(26.97) 


That is, in addition to the A matrix, the B and C matrices must have LBT structure. In addition to the 
desired structure of the system matrices, it is also necessary that the decoupled subsystems represented by 
(Ajj, Bii, Ci) are at least structurally controllable and observable, and that none is further decomposable. 

Because the LBT structure is concerned with the reachability properties of the system, both this struc- 
ture and input and/or output reachability requirements for the subsystems, which are necessary for 
structural controllability and/or observability, can be taken care of by a suitable decomposition scheme 
based on binary operations on the reachability matrix of the system digraph. The requirement that the 
subsystems be dilation free, which is the second condition for structural controllability and/or observabil- 
ity, is of a different nature, however, and should be checked separately after the input-output reachability 
decomposition has been obtained. 

When outputs are of no concern, it is easy to identify all possible acyclic, irreducible, input reachable 
decompositions of a given system. If some of the resulting decoupled subsystems turn out to contain 
dilations (destroying structural controllability), then they can suitably be combined with one or more 
subsystems at a higher level of hierarchy to eliminate the dilations without destroying the LBT structure. 
Provided that the overall system is structurally controllable, this process eventually gives an acyclic, 
irreducible decomposition in which all subsystems are structurally controllable. Of course, dual statements 
are valid for acyclic output reachable decompositions. 

Once an acyclic decomposition into controllable subsystems is obtained, many design problems can be 
decomposed accordingly. An obvious example is the state feedback structure in Equation 26.96. A more 
complicated problem is the suboptimal state feedback design discussed in the section on optimization. 
For the system in Equation 26.97, the test matrix F(|1), with the inclusion of the input coupling terms B;;, 
becomes 

F(Mp) = Fp(Mp) + Fc(Mp) + F¢(Mp), (26.98) 


where Mp = diag{11, 2, ..., Wn}, allowing different |1;’s for $;’s, Fo(Mp) = [11 — 1b; (Qi + K}R;Kj)], 
and Fco(Mp) = [Fj(\1i)] with 


py 1Pi(Ay — ByKj), i> j 


0, fen (26.99) 


Fi (ui) = | 
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From the structure of F(Mp) it is clear that the choice p; = N+!—‘, i EN, results ina negative definite 
F(Mp) for sufficiently small «. This guarantees existence of a suboptimal state feedback control law with 
the degree of suboptimality j. = €'. In practice, it is possible to achieve a much better 1 by a careful 
choice of the weight matrices Q; and Rj. 

In a similar way, acyclic, structurally observable decompositions can be used to design suboptimal 
state estimators, which are discussed below in the context of sequential optimization for acyclic IO 
decompositions. 

To illustrate the use of acyclic IO decompositions in a standard LQG optimization problem, it suffices 
to consider decomposition of a discrete-time system into only two subsystems as 


$1: xi(f+ 1) = Aixi(t) + Bumi(t)+ wilt), 
yilt) = Cuixi(t) + v(t), 

So: X2(t + 1) = A2ix1(t) + Az2xa(t) + Bai uy (t) + Bogua(t) + w2(t), 
y2(t) = Cox) (t) + Co2x2(t) + va(t), 


(26.100) 


with the usual assumptions on the input and measurement noises w; and vj,i = 1,2. Let each subsystem 
be associated with a performance criterion 


T-1 
Ee | fim TS [xP Qxi(t) + son] 2 4=1,2 (26.101) 
T>0o = 
where € denotes expectation. 

The sequential optimization procedure consists of minimizing €J,; and EJ, subject to the dynamic 
equations for the systems S; and (S,,S2), respectively. The first problem has the standard solution 
uy (t) = —K X(t), where Kj is the optimal control gain found from the solution of the associated Riccati 
equation, and x; (ft) is the best estimate of x) (t) given the output information ve = {y1(0),...,y1(t — 1)}. 
The estimate x(t) is generated by the Kalman filter 


X(t + 1) = Ay 1 (t) + Bruy (1) + Lily (f) — ¢1151(0)] (26.102) 


where L, is the steady-state estimator gain. With the control uj} applied to $1, the overall system becomes 


X(t + 1) Ay —ByK,)-LiCy, L1Cy 0 X(t) 0 Ly, (t) 

S:|x(t+1)}= —B Kk, Ai 0 xi(t)|+] O | u(t)+] wilt) 

x2(t +1) —B2K, Az)  Az2_| Lx2(t) By2 w(t) 
(26.103) 


which preserves the LBT structure of the original system. Assuming that both ye and yy = 
{y2(0),...,¥2(t— 1)} are available for constructing the control u} (which is consistent with the idea 
of sequential optimization), the problem reduces to minimization of EJ subject to (103). An analysis of 
the standard solution procedure reveals that the optimal control law can be expressed as 


u3(t) = —Ky1%1(t) — K&(t) (26.104) 


where K = [K2, Ka] is the optimal control gain, and E(t) is the optimal estimate of x(t) = [x7 (t) x Ol, 
given ya and a eae Furthermore, the 2m; + n-dimensional Riccati equation, from which K is con- 
structed, can be decomposed into an n2-dimensional Riccati equation involving the parameters of the 
second isolated subsystem and a Lyapunov equation corresponding to an nz x 2; dimensional matrix. 
This results in considerably simplifying the solution of the optimal control gain. However, the Kalman 
filter for E(t) still requires the solution of an (n; + nz)-dimensional Riccati equation. 

Other sequential optimization schemes based on various information structure constraints can be 
analyzed similarly; for details, see [30]. 
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26.8.3 Nested Epsilon Decompositions 


Epsilon decomposition of a square matrix M is concerned with transforming M by symmetric row and 
column permutations into a form 
P™MP = Mp+«Mc (26.105) 


where Mp is block diagonal, and € is a prescribed small number [27]. The problem is equivalent to 
identifying the connected components of a subgraph D‘ of the digraph D associated with M, which is 
obtained by deleting all edges of D corresponding to those elements of M with magnitude smaller than e. 
All of the vertices of a connected component of D‘! appear in the same connected component of D® for 
any €2 < ey. Thus one can identify a number of distinct values €; > €2 >--- > €x such that 


P' MP = (...((My +€:M1) + €2Mo) +--- +€KMx); (26.106) 


which is a nested epsilon decomposition of M as illustrated in Figure 26.3. 

As seen from the figure, a large € results in a finer decomposition than a small € does. Thus the 
choice of € provides a compromise between the size and the number of components and the strength 
of the interconnections among them. A nice property of nested epsilon decompositions is that once the 
decomposition corresponding to some €, is obtained, the decomposition corresponding to €,4 can be 
found by working with a smaller digraph obtained by condensing D“ with respect to its components. 

An immediate application of the nested epsilon decompositions is the stability analysis of a large scale 
system via vector Lyapunov functions, where the matrix M is identified with the matrix A of the system 
in Equation 26.6. Provided the subsystems resulting from the decomposition are stable, the stability of 
the overall system can easily be established by means of the aggregate matrix W in Equation 26.49, whose 
off-diagonal elements are of the order of €. 

The nested epsilon decomposition algorithm can also be applied with some modifications to decompose 
a system with inputs as 


N 
x; = Ayx; + Biyuj +e Ags + Byuj), ieN. (26.107) 

Ai 
If each decoupled subsystem identified by a pair (Ajj, Bj) is stabilized by a local state feedback of the form 
uj = —Kjx;, i€ N, with the local gains not excessively high, then the closed-loop system preserves the 


FIGURE 26.3 Nested epsilon decompositions. 
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weak-coupling property of the open-loop system, providing an easy way to stabilize the overall system. 
The same idea can also be employed in designing decentralized estimators [30] based ona suitable epsilon 
decomposition of the pair (A, C). 


26.8.4 Overlapping Decompositions 


Consider a system 


S: x(t) = Ax(t) (26.108) 


with an ”-dimensional state vector x. Let columns of the matrix V € R’*” form a basis for an n- 
dimensional A-invariant subspace of R”, and let A be the restriction of A to ImV ~ R", that is, AV = VA. 
Then the smaller order system 


S: x(t) = Ax(t) (26.109) 


is called a restriction of §. Conversely, starting with the system $, one can obtain an expansion § of $ 
by defining A = VAV! + M, where V® is any left inverse of V, and M is any complementary matrix 
satisfying MV = 0. The very definition of a restriction implies that $ is stable if $ is. 

In many problems associated with large scale systems, it may be desirable to expand a system S to a 
larger dimensional one which possess some nice structural properties. The increase in dimensionality of 
the problem may very well be offset by the nice structure of the expansion. As an example, consider a 
system S with 


Ai Aj2 | €A13 
A= €A21 A22 €A3 (26.110) 
€A31 | A322  A33 


where € is a small parameter. Letting 


Tr 0 0 
5.0 
V= 26.111 
0 b 0 ( ) 
008 
where J; denotes an identity matrix of order ng, one obtains an expansion S$ with 
An Aiz| 0 €Ax3 
~ A A 0 A 
Aa See =f23 (26.112) 


€A21 0 Ax €A3 
€A31 0 | Az. A33 


Since § has an obvious decomposition into two weakly coupled subsystems, one can take advantage of 
this structural property in stability analysis, which is not available for the original system S. 

One can easily notice from the structure of V in Equation 26.114 that the expansion S of $ is obtained 
simply by repeating the equation for the middle part x2 of the state vector x = [x] x} x]". In some 
sense, X2 is treated as common to two overlapping components x; = [xf we JF and % = [xf ca JF of x. 
Thus the partitioning of the A matrix in Equation 26.113 is termed the overlapping decomposition. 

Although the expansion matrix V can be any matrix with full column rank, if it is restricted to contain 
one and only one unity element in each row (which corresponds, as in the case above, to repeating 
some of the state equations in the expanded domain), then one can develop a suitable graph-theoretic 
algorithm to find the smallest expansion which has a disjoint decomposition (into decoupled or e-coupled 
components) with the property that no component is further decomposable. 
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The idea of overlapping decompositions via expansions can be extended to systems with inputs. A 
system ; 
S: x(t) = Ax(t) + Bu(t) (26.113) 


is said to be an expansion of 
S: x(t) = Ax(t) + Bu(t) (26.114) 


if B= VB in addition to AV = VA. Consider the optimal control problems of minimizing the perfor- 
mance criteria 


j= / Ex?) Qx() +u' (t)Ru(t)] dt 
0 


ee (26.115) 
7= [wT + wTORUE At 
0 
associated with $ and S. The optimal solutions are 
u(t) = —Kx(t), and u(t) = —Kx(t), (26.116) 
respectively, resulting in closed-loop systems 
§ : x(t) = (A — BK)x(t), 
(26.117) 


S38) =a BOR. 


Thus, § is a restriction of § if (A — BK) V = V(A— BK), or equivalently, if K =KV. The last condition 
is satisfied if Q and Q are related as Q = V! QV, in which case the optimal cost matrices are also related 
as P = V’ PV. This analysis shows that, if the cost matrices Q and R of the expanded system are chosen 
to be block diagonal with diagonal blocks associated with the decoupled expanded subsystems, then its 
optimal (in case of complete decoupling) or suboptimal (in case of weak decoupling) solution can be 
contracted back to an optimal or suboptimal solution of the original system with respect to a suitably 
chosen performance criterion. 
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Further Reading 


There is a number of survey papers on decentralized control and large scale systems [3,14,24]. The 
books on the subject are [10,15,19,29,31]. For a comprehensive treatment of decentralized control theory, 
methods, and applications, with a large number of references, see [30]. 

For further information on vector Lyapunov functions and stability analysis of large scale intercon- 
nected systems, see the survey papers [22,33], and books [16,17]. 

Adaptive decentralized control has been of widespread recent interest, see [2,9,21,23,28,30,36]. 
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Robustness of decentralized control to both structured and unstructured perturbations has been one of 
the central issues in the control of large scale systems. For the background of robustness issues in control, 
which are relevant to decentralized control, see [18,30]. For new and interesting results on the subject, 
see [4-6,8]. 

There is a number of papers devoted to design of decentralized control via parameter space optimiza- 
tion, which rely on powerful convex optimization methods. For recent results and references, see [11]. 

Overlapping decentralized control and the inclusion principle are surveyed in [30]. Useful extensions 
were presented in [13]. The concept of overlapping is basic to reliable control under controller failures 
using multiple decentralized controllers [30]. For more information about this area, see [7,12,32,35]. 

In a recent development [25], it has been shown how optimal decentralized control of large scale inter- 
connected systems can be obtained in the classical optimization framework of Lagrange. Both sufficient 
and necessary conditions for optimality are derived in the context of Hamilton-Jacobi equations and 
Pontryagin’s maximum principle. 
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27.1 Introduction 


Multi-input/multi-output systems are usually difficult for human operators to control directly, since 
changing any one input generally affects many, if not all, outputs of the system. As an example, consider 
the vertical landing of a vertical take off and landing jet or of a lunar landing rocket. Moving to a desired 
landing point to the side of the current position requires tilting the thrust vector to the side; but this 
reduces the vertical thrust component, which was balancing the weight of the craft. The aircraft therefore 
starts to descend, which is not desired. To move to the side at a constant height thus requires smooth, 
simultaneous use of both attitude control and throttle. It would be simpler for the pilot if single control 
existed to do this maneuver; hence the interest in control methods that make the original system behave in 
a way that is easier to control manually. One example of such technique is when a compensator is sought 
that makes the compensated system diagonally dominant. If this can be achieved, it is then possible to 
regard the system as, to first order, a set of independent single-input/single-output systems, which is far 
easier to control than the original plant. Another approach is that of decoupling, where the system transfer 
matrix is made to be exactly diagonal. Each output variable is therefore affected by only one input signal, 
and each input/output pair can then be controlled by an easier-to-design single-input/single-output 
controller or manually by a human operator. 

This chapter studies the problem of making the transfer function matrix of the system diagonal using 
feedback control and, in particular, state feedback, state feedback with dynamic precompensation, and 
constant output feedback control laws. This problem is referred to as the dynamical decoupling problem, 
as it leads to a compensated system where the input actions are decoupled; it is also called a noninteracting 
control problem for similar reasons. Stability is an important issue and it also is examined here. Conditions 
for decoupling with stability and algorithms to determine such control laws are described. The problems 
of block diagonal or triangular decoupling are also addressed. They are of interest when full diagonal 
decoupling using a particular form of feedback control, typically state feedback, is not possible. Note 
that the approach taken in this chapter follows the development in [14]. Static decoupling is also briefly 
discussed; references for approximate diagonal decoupling are provided in “Further Reading.” 
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27.1.1 Diagonal Decoupling 


Diagonal decoupling of a system with equal numbers of inputs and outputs is the most straightforward 
type of problem in the field of noninteracting control. The goal is to apply some form of control law to 
the system so as to make the ith output of the closed-loop system independent of all but the ith closed- 
loop input signal. Each output can then be controlled by a dedicated simpler single-input/single-output 
controller, or by a human operator. The main questions to be answered when investigating diagonal 
decoupling of a given system are 


¢ Can it be decoupled at all? 
¢ Ifso, what form of controller is required to achieve this? 


Three classes of controllers that are customarily considered are 


1. Constant output feedback u = Hy + Gr, where the output y of the system is simply multiplied by 
a constant gain matrix H and this is fed back as the control signal u, with r the new external input 
to the system and G a constant gain matrix 

2. Linear state feedback u = Fx + Gr, where the control signal consists of a constant matrix F multi- 
plying the internal state variable vector x of the system 

3. State feedback plus precompensation, where a feedforward dynamic control system is added to 
the state feedback controller 


Note that the compensator in class 3 corresponds to dynamic output feedback, where the input and output 
signal vectors r and y are multiplied by dynamic transfer function gain matrices rather than constant ones. 

The problem of diagonally decoupling a square system was the first decoupling question to be studied, 
and it can be answered in a fairly straightforward fashion. First of all, diagonal decoupling by state feedback 
plus precompensation, or by dynamic output feedback, amounts to finding a transfer matrix that, when 
the open-loop transfer matrix is multiplied by it, produces a diagonal closed-loop transfer matrix. This 
problem is therefore closely related to the problem of finding an inverse for the open-loop plant. As a 
result of this, any square plant that has a full rank transfer matrix can be diagonally decoupled by this 
type of control. This result was proved by Rekasius [10]. A system that does not satisfy this condition 
does not have linearly independent outputs, so it follows that it is impossible to decouple by any form 
of controller. It is of great practical interest to establish whether a given plant can actually be decoupled 
by a simpler type of controller than this. Falb and Wolovich [3] established the necessary and sufficient 
condition under which diagonal decoupling by state feedback alone is possible. This condition, which 
can be easily tested from either a state-space or a transfer matrix model of the plant, can be expressed as 
follows. 

A square system can be diagonalized by state feedback alone if and only if the constant matrix B* is 
nonsingular, where this matrix is defined below first from the state-space and then from the transfer 
matrix description of the system. 

State-space representation. Let the given system be x = Ax + Bu, y = Cx + Duin the continuous-time 
case, or x(k + 1) = Ax(k) + Bu(k), y(k) = Cx(k) + Du(k) in the discrete-time case; let A, B,C, Dben x n, 
n X p, p X n, p x p real matrices, respectively; and assume for simplicity that the system is controllable and 
observable. Then the p x p matrix B* is constructed as follows: If the ith row of the direct feedthrough 
matrix D is nonzero, this becomes the ith row of the constant matrix B*. Otherwise, find the lowest 
integer, f;, for which the ith row of CA‘~'B is nonzero. This then becomes the ith row of the constant 
matrix B*. 

Transfer matrix representation. Let T(s), with s the Laplace transform variable, be the p x p transfer 
function matrix of the continuous-time system; or T(z), with z the Z-transform variable, be the transfer 
function matrix of the discrete-time system. Let D(s) [or D(z)] be the diagonal matrix D(s) = diag (si ) 
where the nonnegative integers {fj} are so that all rows of lims_,o D(s)T(s) are constant and nonzero. 


Decoupling 27-3 


This limit matrix is B*; that is, 
lim D(s)T(s) = B* (27.1) 
S=> 00: 


The integers {f;} are known as the decoupling indices of the system. They can be determined from either 
the state-space or the transfer function descriptions as described above; note that f; = 0 corresponds to 
the ith row of D being nonzero. In either case, of course, the resulting matrix B* is the same. It should 
be noted that systems will generically satisfy the decoupling condition; that is, if all entries of the A, B, C 
(and D) matrices are chosen at random, the resulting B* will have full rank. Diagonal decoupling by state 
feedback is therefore likely to be feasible for a wide variety of systems. 


Example 27.1: 


0 1 0 0 O 
A=] 0 0 1], B=] 0 Of, c=(3 : HE o=(5 
-—6 -11 -6 -1 2 


This gives f; = 1, fg = 3, and BX = a . This matrix is clearly singular; therefore, the system 


—2 
cannot be decoupled by state feedback. 
Example 27.2: 
1 2 
= Ky s+1 
teh 4 85 
s+3 s+4 
This gives B* = 0 ) , with decoupling indices f; = 1, fy = 0. This system can therefore be diago- 


nally decoupled by state feedback. 


Example 27.3: 
1 2 
= Ss s+1 
Tiel ge * 4g 
s+3 s+4 


The same as previously, but with the (2,2) entry divided by s. We now obtain B* = (; s) , with 


decoupling indices f; = 1, f) = 1. B™ is now singular, so this system cannot be diagonally decoupled 
by state feedback alone. 


27.1.2 Diagonal Decoupling with Internal Stability 


A question of great practical interest is whether the closed-loop system that is obtained after decoupling 
can be made stable. It can be shown constructively (for instance, by use of the algorithm given below) 
that all of the poles that are evident from the diagonal closed-loop transfer matrix can be assigned any 
desired values. The question therefore becomes: Can the closed-loop system be made internally stable, 
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where there are no “hidden” cancellations between unstable poles and zeros? Such unstable modes are 
particularly dangerous in practice, as they will not be revealed by an examination of the transfer matrix. 
However, the hidden unstable state behavior they represent will very likely cause problems, such as 
burnout of internal electronic components of the system. It was shown by Gilbert [5] that a given plant 
may indeed have hidden fixed modes when it is diagonally decoupled by state feedback, with or without 
precompensation. Wolovich and Falb [15] then showed that these modes are the same for both cases; 
furthermore, they are a subset of the transmission zeros of the plant. In fact, they are those transmission 
zeros z; that do not make any of the rows of the transfer matrix T(s) equal to zero when evaluating T(z;); 
they are called diagonal coupling zeros. Thus, any plant with square, full-rank transfer matrix for which all 
the diagonal coupling zeros are in the left half-plane can be diagonally decoupled with internal stability 
by state feedback plus precompensation; or by state feedback alone if B* is nonsingular. Therefore, there 
will never be any problems with internal stability when decoupling a minimum-phase system, as all of its 
transmission zeros are in the left half-plane. 

An algorithm to diagonally decouple a system, when B* has full rank, using state feedback is now 
presented. This algorithm is based on a procedure to obtain a stable inverse of a system that is described 
below. This procedure is applied to the system D(s)T(s) = T(s), where D(s) = diag(sf') as in Equation 27.1, 
that can be shown to have a state-space realization {A, B, C, D}. In fact D = B*, which is assumed to have 
full rank p; and this implies that a proper right inverse of the system T(s) exists. Here it is assumed that 
the system has the same number of inputs and outputs, and this simplifies the selection of F, G as in this 
case they are unique; see the algorithm for the inverse below for the nonsquare case. In particular, if the 
state feedback u = Fx + Gr with 


F=-(B*)'C, G=(B*)7! (27.2) 


is applied to the system x = Ax + Bu, y = Cx + B*u, then it can be shown that Tr.¢(s) = D(s)Tr,e(s) = Ip. 
This implies that if the state feedback u = Fx + Gr with F, G as in Equation 27.2 is applied to the given 
system x = Ax + Bu, y = Cx + Du with transfer matrix T(s), then 


Tr,G(s) = D~'(s) (27.3) 


which is diagonal with entries s~/'. Note that here the state feedback matrix F assigns all the n closed- 
loop eigenvalues at the locations of the n zeros of T(s); that is, at the zeros of T(s) and of D(s). The 
closed-loop eigenvectors are also appropriately assigned so the eigenvalues cancel all the zeros to give 
D(s)Tp,G(s) = Ip. This explains the control mechanism at work here and also makes quite apparent the 
changes necessary to ensure internal stability. Simply instead of D(s) use D(s) = diag{p;(s)} with p;(s) 
stable polynomials of degree s/‘; that is, p;(s) = s/ + lower-degree terms. Then it can be shown that 
lims—+o0 D(s)T(s) = B* and that (A, B, C, B*} is a realization of D(s)T(s) = T(s). State feedback with 


F=—-(B*)!C, G=(B*)7! (27.4) 


gives 
Tr,g(s) = D~\(s) = diag{p; '(s)} (27.5) 


which is stable. Note that in this case the closed-loop eigenvalues are at the assumed stable zeros of T(s) 
and at the selected stable zeros of the polynomials p(s), i=1,...,p. 


Decoupling 27-5 


Example 27.4: 
1 
— 0 
Let T(s) = 1 4 
s(s—1) s—1 
Here 


lim, D(s)T(s) = lim, diag(s, s)T(s) = (; a = B*, 


Since B™ has full rank, the system can be decoupled using state feedback u = Fx + Gr. The system 
has one transmission zero at —1 and there are no diagonal coupling zeros, so it can be decoupled 
s+] 0 

0 s+2 
of T(s) = D(s)T(s) is {A, B, C, B*} where 


with internal stability. Let D(s) = ( ) .A minimal (controllable and observable) realization 


a (I 
A=|0 0 ol, B= 
=f 00-4 


o=- oO 
-0o°oO 
(ay) 
ll 

—~ 
wo 
—- NY 
lo 
w 

SS 


In view now of Equations 27.4 and 27.5, for 
F=-(8*)'C= (a eee ) 


and 


s+2 


The closed-loop eigenvalues are in this case located at the transmission zero of the plant at —1 and 
at the selected locations —1 and —2, the poles of D(s)7). Note that it is not necessary to cancel the 
transmission zero at —1 in order to decouple the system since it is not a coupling zero; it could appear 
as a zero in the decoupled system instead. To illustrate this, consider Example 27.5 where T(s) is the 
same except that the zero is now unstable at +1. 


Example 27.5: 
s—1 
=F 0) 
Let T(s) = i 1 where again 
s(s—1) s—1 


lim, D(s)T(s) = lim, diag(s, s)T(s) = (; Ey = BX, 


Since BX has full rank, the system can be decoupled using state feedback. Since there are no diagonal 


coupling zeros, the system can be decoupled with internal stability. Write T(s) = (' 6 i ) Tn(s) 
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2 
and apply the algorithm to diagonally decouple Tyy(s). Now Dy(s) = G : and take 


By ee ra 


A minimal (controllable and observable) realization of Tjy(s) = Dy(s)Tyy(s) is {A, B, Cn, Bx} where 


and 


= 


—______ 0 
(Tye.g(s) = By(sy- 1 =| StAG+3) | 
0 


s+1 
If now this state feeback is applied to the minimal realization {A, B, C} of T(s)—note that A, B are the 
s—1 
ee 0 
same as above—then TF g(s) = . 6 : ‘) By — | (s+2)(s+3) 1 . Note that the unstable 
0 Pani 


s+1 
noncoupling transmission zero at +1 appears on the diagonal of the compensated system; the 


closed-loop eigenvalues are at the arbitrarily chosen stable locations —1,—2 and —3. 


Algorithm to Obtain a Proper Stable Right Inverse Using State Feedback 


Let x = Ax + Bu, y = Cx + Du with A, B,C, D nx n,n x m, p x n, p x m real matrices, respectively, and 
assume that the system is controllable and observable. Let T(s) be its transfer function matrix. It is known 
that there exists a proper right inverse Tr(s), such that T(s)TR(s) = Ip, if and only if rankD = p. If, in 
addition, all the zeros of T(s) (that is, the transmission zeros of the system) are stable, then a stable right 
inverse of order n can be constructed with k(< n) of its poles equal to the k stable zeros of T(s) with the 
remaining n — k poles arbitrarily assignable. This can be accomplished as follows: 

Let Teg = F[s] —(A+ BF)|~'BG + G where F, G aren x m,m x p> respectively, and note that 


T(s)Teq(s) = [C(sI — A)~'B + D][F[sI — (A+ BF)]~"'BG + G] 
= (C+ DF)[sI — (A+ BF)]-'BG+ DG = Tp,¢(s) 


(27.6) 


which is the transfer matrix one obtains when the state feedback control law u = Fx + Gr is applied to 
the given system. Note that the second line of Equation 27.6 results from application of a well-known 
formula for the matrix inverse. If now F, G are such that 


C+DF=0, DG=h (27.7) 


then Tp,g(s) = Ip and Te, is a proper right inverse Tp(s). The additional freedom in the choice of F when 
p < mis now used to derive a stable inverse; when p = m, F, G are uniquely determined from F = —D~!C, 
G=D". 
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If the nonsingular m x m matrix M is such that DM = (Ip 0), then C+ DF = C+ DMM7!F=C+ 


FR) | ee fe 
(Ip 0) (71) = 0 from which F = M ( p 


) with F, arbitrary. Also, from DG = DMM~!G = Ip, G= 
2 


I im = iin? oh = 
M (2) with G» arbitrary. The eigenvalues of A+ BF = A+ BM ( i) =A+(B, Bo) ( i) =A- 
2 2 2 


B,C + BoF, are the poles of Tp(s). It can be shown that the uncontrollable eigenvalues of (A — B,C Bp) 
are exactly the (k) zeros of the system; they cannot be changed via F). The remaining n — k controllable 
eigenvalues can be arbitrarily assigned using F2. In summary, the steps to derive a stable proper inverse 
are 


Step 1. Find nonsingular m x m matrix M such that DM = (I, 0). 

Step 2. Calculate (By Bo) = BM, and A — BC. 

Step 3. Find By that assigns the controllable eigenvalues of (A — B,C Bo) to the desired locations. The 
remaining uncontrollable eigenvalues are the stable zeros of the system. 


Step 4. 
—C I —C I 
A+BM(~),BM(2),M[.~),mM(#? 27.8 
[a+ (Gs) (2) (i) (2)} oe 


where Gy a (m-— p) x p arbitrary real matrix, is a stable right inverse. 


Teq(s) above is the open-loop equivalent to the state feedback control law. In view of Equations 27.6 and 
27.7 the above algorithm selects F, G in a state feedback control law u = Fx + Gr so that the closed-loop 
transfer matrix Tp,¢(s) = Ip and the closed-loop system is internally stable; that is, all the n eigenvalues 
of A+ BF are stable. Note that when p = m, then F, G are uniquely given by F= —D~!C, G= D7}; the 
eigenvalues of A + BF are then the n zeros of the system. In this development of stable inverses via state 
feedback, the approach in [1] was taken; see also [7,12]. 

In order to implement decoupling by state feedback in practice, it is often necessary to estimate the 
internal state variables by means of an observer. Certain plants can be decoupled by constant output 
feedback, avoiding the need for an observer. The necessary and sufficient conditions under which this is 
possible were proved by Wolovich [18]: it is that B* not only be nonsingular, but also that the modified 
inverse transfer matrix B*T~'(s) have only constant off-diagonal elements. This appears to be a very 
stringent condition, so diagonal decoupling by means of constant output feedback is not likely to be 
possible for any but a relatively small class of plants. This is in clear contrast with the state feedback case, 
as mentioned previously. If diagonal decoupling by output feedback is possible, any gain matrix H that 
achieves it must give all off-diagonal entries of B*H equal to those of B* T~1(s). It can therefore be seen 
that any constant matrix of the form (B*)~!Z can be added to H, where Z is an arbitrary diagonal matrix, 
and still give a gain matrix that satisfies the required condition. There is thus a small amount of controller 
design freedom available, which can be used, for instance, to assign closed-loop poles to some extent. 
However, it does not appear possible to quantify this pole-placement freedom in any straightforward 
manner. 


27.1.3 Block Decoupling 


If diagonal decoupling by linear state feedback is not possible, an alternative to applying precompensation 
may still exist. It may be possible to use state feedback, or perhaps even output feedback, to reduce the 
system to a set of smaller subsystems that are independent; that is, decoupled. Controlling each of these 
small systems can then be performed in isolation from all the others, thus reducing the original plant 
control problem to several simpler ones. This is the idea behind block decoupling, where the goal is to 
transform the plant transfer matrix to one that is block diagonal rather than strictly diagonal. For square 
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plants, each of these k diagonal blocks will also be square: the ith will be taken to have p; inputs and p; 
outputs, with )> p; = p. 

One question associated with block decoupling can be answered immediately: namely, any plant with 
nonsingular transfer matrix can be block decoupled by linear state feedback plus precompensation. This 
follows from the fact that any such system can be diagonally decoupled by this form of compensation 
and so is trivially of any desired block diagonal form. The two types of compensation that have to be 
addressed here are therefore state feedback and constant output feedback. 

If we are interested in block decoupling a given system by state feedback, this implies that it was not 
fully diagonalizable by state feedback. Hence, the matrix B* must have been singular. As the inverse of 
this matrix played a significant role in the development of diagonal decoupling compensators, it seems 
likely that overcoming this singularity may lead toward designing block decoupling compensators for 
systems that cannot be diagonalized by state feedback. An equivalent way of stating that B” is singular is 
to note that, although all rows of lims—.o9 D(s)T(s) are certainly finite and nonzero, some of these rows 
must have been linearly dependent on the preceding ones. Suppose the ith row is one such. It is then 
possible to add multiples of rows 1, ...,i— 1 to rowi in order to zero out the ith row in B’; that is, to make 
what had been the leading coefficient vector of this row of D(s)T(s) zero. If the new leading term in this 
row is now of order s~*, multiplying the row by s* yields a new finite and nonzero limit as s goes to infinity. 
If this row vector is independent of the preceding ones, we have now increased the rank of the modified 
B* -like matrix; if not, the same process can be repeated until successful. This basic procedure leads to the 
following definition, which has proved to be very useful for studying block decoupling problems. 

The interactor X7(s) of T(s) is the unique polynomial matrix of the form X7(s) = H(s)A(s), where 
A(s) = diag(s!') and H(s) is a lower triangular polynomial matrix with 1s on the diagonal and the nonzero 
off-diagonal elements divisible by s, for which 


lim X7(s)T(s) = Kr (27.9) 
S—>0O 
is finite and full rank. The interactor can be found from the transfer matrix of the system [16]; from a 


state-space representation [4]; or from a polynomial matrix fraction description for it [13]. The basic 
procedure can be illustrated by applying it to two examples discussed previously. 


Example 27.6: 
1 2 
TOSI an, “ae 
s+3 s+4 


2 


1 
test i, 2 
This gives BX = (; 3 


). with decoupling indices f; = 1, f = 0. B* is nonsingular, so it satisfies 


the definition of the desired matrix Kr. Thus, Kr = B* = (; 3 


: a here, and X7(s) = diag(s", s2) = 


Example 27.7: 
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Bx = ¢ a , which is singular, with decoupling indices f; = 1, f2 = 1. Subtracting 4 times row 1 of 


diag(s‘i)T(s) from row 2 eliminates the linearly dependent leading coefficient vector. The resulting 
lower-degree polynomial row vector can then be multiplied by s, so as to again obtain a finite limit 
as s goes to infinity. We then have 


a 1 0 1 O\/s O 
nw =(4 ) & 7 ic *) 
2s 

seat 
—12s —24s? 
s+3  (s+1)(s+4) 


1 
T(s)= 


Unfortunately, Ty (s) has limit as s goes to infinity of 


1 2 
—12 —24)' 


which is still singular. We therefore have to repeat the procedure, this time adding 12 times row 1 to 
row 2 to eliminate the leading coefficients and multiplying the resulting row by s to give it a finite 
2s 
1 


te ._ {1 0 1 O\~ = s+] P re 
limit. We then obtain (; :) e ) T1(s) = 36s 9652 , which has limit as s goes 


s+3 (s+ 1)(5+4) 


ee ) for this system. The interactor 


36 96 


toinfnity of ( 36. 06 


aes ) . This is clearly nonsingular, so Ky = ( 


is then 


1 0 s 0 
—As2 +125 1 0 3 


which is of the desired form H(s)A(s). 


It can be seen that, if B* is nonsingular, no additional row operations are needed to modify it to give 
the nonsingular Kr. Thus, in this case B* = Ky and D(s) = X7(s). But we already know that diagonal 
decoupling by state feedback is possible in this case; that is, diagonalization by state feedback is possible 
if and only if the interactor of the system is diagonal. This suggests the following general result. 

A square system can be block decoupled by state feedback if and only if its interactor is of this same block 
diagonal structure. 

A proof of this result is based on the fact that state feedback matrices F, G can always be found that 
make the closed-loop transfer matrix equal to the inverse of its interactor; see the algorithms discussed 
previously and [2,6]. Thus, if this matrix is block diagonal, so is the closed-loop transfer matrix. The state 
feedback that achieves this form can be found in an analogous manner to the state feedback matrices 
determined above that diagonally decouple the system. 

Note that the structure algorithm of Silverman [11] is quite closely related to the interactor. This method 
determines a polynomial matrix X(s) such that lims_,. X(s)T(s) is finite and nonsingular; however, X(s) 
is not of any particular structure, unlike the interactor. This makes X7(s) better suited to obtaining clear 
block decoupling results. 
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Another question that generalizes naturally from the diagonal case is that of stability. The only fixed 
modes when diagonalizing were the diagonal coupling zeros, which were all zeros of the original plant 
that were not also zeros of any of the rows of the plant transfer matrix. In the case of block decoupling, the 
only fixed poles are the block coupling zeros, which are all zeros of the plant that are not also zeros of one 
of the (p; x m) row blocks of T(s). As in the diagonal case, these zeros must be cancelled by closed-loop 
poles in the decoupled transfer matrix, so creating unobservable modes; all other poles can be assigned 
arbitrarily. 

Finally, it may be possible to achieve block decoupling by the simpler constant output feedback com- 
pensation. It can be shown that the interactor also allows a simple test for this question. In fact, block 
decoupling by constant output feedback is possible if and only if the interactor of the system is block 
diagonal and the modified inverse system KT ‘(s) has only constant entries outside the diagonal blocks. 
The output feedback gain matrix H that achieves block decoupling is such that K7H is equal to the con- 
stant term in Kr T~ ‘(s) outside the diagonal blocks. This is very similar to the diagonal decoupling result. 
As there, a certain degree of flexibility exists in the design of H, due to the fact that the diagonal blocks 
of KrT~1(s) are essentially arbitrary; this can be used to provide a small amount of pole assignment 
flexibility when decoupling. 


27.1.4 Decoupling Nonsquare Systems 


The previous development has been primarily for plants with equal numbers of inputs and outputs. 
Plants that are not square present additional complications when studying decoupling. For instance, if 
there are more outputs than inputs, it is clearly impossible to assign a separate input to control each output 
individually; diagonal decoupling in its standard form is therefore not feasible. Similarly, decoupling the 
system into several independent square subsystems is also impossible. On the other hand, plants with 
more inputs than outputs present the opposite difficulty: there are now more input variables than are 
required to control each output individually. 

Fortunately, the classical decoupling problem can be generalized in a straightforward fashion to cover 
nonsquare plants as well as square ones. In view of the preceding remarks, it is clear that systems with 
more outputs than inputs (p > m) must be analyzed separately from those with more inputs than outputs 
(p < m). The former case leads to decoupling results that are barely more complicated than those for the 
square case; the additional design freedom available in the latter case means that conditions that were 
necessary and sufficient for p = m become only sufficient for p < m. 

Taking the case of more inputs than outputs (p < m), the following results can be shown to hold for 
diagonal decoupling. First, any such plant that is right-invertible (that is, for which the transfer matrix 
is of full rank, p) can be decoupled by state feedback plus precompensation; this follows from the close 
connections between this type of decoupling control law and finding a right inverse of the system. If we 
restrict ourselves to state feedback, two sufficient conditions for diagonal decoupling can be stated. First, 
the plant can be diagonalized by state feedback if its matrix B* is of full row rank, p. This is extremely 
easy to test, but can be somewhat conservative. A tighter sufficient condition is as follows: The plant can 
be diagonalized by state feedback if a constant (m x p) matrix G can be found for which the B* matrix of 
the square-modified transfer matrix T(s)G is nonsingular. 

It may be thought that these two sufficient conditions are identical. To see that they are not, con- 

2 
sider the following simple example: T(s) = ( ¢ a /s? ve 1) has B* = € : i) , which has 
only rank 1. The first sufficient condition for diagonal decoupling is therefore violated. However, post- 
multiplying T(s) by the matrix 
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gives 
1/0 1 


which clearly has nonsingular B* matrix of . The role of the G matrix is basically to cancel those 


0 1 
(; 1 
higher-power terms in s in T(s) that give rise to linearly dependent rows in B*; in the example, the first 
column of G, (1 0 — 1)", can be seen to be orthogonal to the repeated row vector [1 0 1] in the original 
B*. Lower-power terms in T(s) then become the leading terms, so their coefficients contribute to the 
new B*; these terms may well be independent of the first ones. An algorithm that goes into the details of 
constructing such a G, if it exists, for any right-invertible T(s) is given by Williams [13]. 

Very similar results apply to the problem of block decoupling a system with more inputs than outputs 
(p < m) by means of state feedback. The more conservative sufficient condition states that the plant can 
be block decoupled if its interactor matrix has the desired block diagonal structure. This can then be 
tightened somewhat by proving that the plant T(s) can be block decoupled if there exists some (m x p) 
constant matrix G that has interactor of the desired block diagonal form. Furthermore, the algorithm 
described previously for block decoupling of square plants can be applied equally in this case, either to 
T(s) or T(s)G. The only distinction of significance between the square case and p < mis that the algorithm 
was proved to use decoupling precompensation of lowest possible order in the square case; for nonsquare 
plants, minimality of this order cannot be proven. 

In the case of plants with more outputs than inputs (p > m), the main complication is in modifying the 
definition of a “decoupled” closed-loop structure. Once this is done, the actual technical results are rather 
straightforward. As already noted, it is no longer possible to assign a single input to each individual output, 
as is required in the classical diagonal decoupling problem. The closest analog to this problem is one where 
the closed-loop system is decoupled into a set of m independent single-input/multi-output subsystems; 
each closed-loop control input influences a set of outputs, but does not affect any of the others. Similarly, 
it is not possible to assign equal numbers of independent inputs and outputs to each decoupled subsystem, 
as holds for square block decoupling. What we must do instead is to define decoupled subsystems that 
generally have more outputs than inputs; that is, they are of dimensions p; x mj, where p; < mj; of course, 
Le pi=pPsdimj=m. 

It can be shown that a very simple rank condition on the plant transfer matrix determines whether or 
not these decoupling problems have a solution. The simplest question to answer is whether the desired 
decoupled structure is achievable by means of a combination of state feedback and precompensation. The 
test is as follows: 

Take the p; rows of the plant transfer function corresponding to the outputs that are to be assigned to 
the ith decoupled subsystem. If this p; x m transfer matrix has rank mj;, and this holds for each i, then the 
plant can be decoupled into p; x mj subsystems by means of state feedback plus precompensation. 

The significance of this result is easiest to see for the special case where m; = 1 for each i, the closest 
analog to diagonal decoupling for systems with p > m. If decoupling is to be possible, we must have that 
each p; x m transfer matrix of the ith subsystem is of rank 1. This implies that the rows of this transfer 
matrix are all polynomial multiples of some common factor row vector. In other words, the p; outputs of 
this subsystem are all made up of combinations of derivatives of a single “underlying” output variable. 
Similarly, decoupling into p; x m; subsystems is possible if and only if the p; outputs making up the ith 
subsystem are actually made up of some combinations of m; “underlying” output variables. 

In practice, applying these rank conditions to the plant transfer matrix dictates what block dimensions 
are possible as closed-loop block decoupled structure. They also show which outputs must be taken as 
members of the same decoupled subsystem. For instance, if we wish to achieve p; x 1 decoupling and 
two rows of the plant transfer matrix are linearly dependent, the corresponding outputs must clearly be 
placed in the same subsystem. 
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But this approach also has one further very important implication. Consider a system that satisfies these 
submatrix rank conditions. If we take the m; “underlying” output variables for each of the r subsystems, 
write down the corresponding mj; x m transfer matrix, and then concatenate these, we obtain a new 
m Xx m transfer matrix, denoted by T,,(s). It can then be shown (see [13]) that a controller will decouple 
T(s) into p; x mj; blocks if and only if it also decouples T;,(s) into square m; x mj; blocks. We can therefore 
take all of the decoupling results derived previously for square plants and use them to solve the problem 
of decoupling systems with more outputs than inputs. In particular, T(s) can be decoupled into p; x m; 
blocks by state feedback if and only if it satisfies the submatrix rank conditions and the interactor matrix 
of Tn(s) is mi x mj; block diagonal. Also, it can be shown that T,,(s) has precisely the same zeros as T(s). 
The two systems therefore clearly also have the same coupling zeros, so the fixed poles when decoupling 
T(s) are the same as the fixed poles when decoupling T;,(s). Finally, decoupling by means of output 
feedback can also be studied by applying the existing results for square systems to the associated T;,(s). 


Example 27.8: 


The state-space model 


0 0 1 1 0 
A=!1 01], B=]{1 Oo], C=h 
1 1 =#0 1 1 
has transfer matrix 
1 5(s+ 1) s 
T(s) = ———~——- ] (s +1), s +1 
Pr 
(s+ 1)(s* —s—1) (412 52 


Clearly, the first two rows are linearly dependent, so this system can be decoupled into the block 


x 0 
diagonal form | x 0] by state feedback plus precompensation. In fact, the associated invertible 
O x 


transfer function for this system is 


1 $A 
eG ielasa (ue 2): 


2 
which has interactor e 7 diagonal is Kr= ( ") Thus, block decoupling is actually 


possible for this system using state feedback alone. 

As a final point on general block decoupling, note that this problem can also be studied using 
the geometric approach; see [19]. This state-space technique is based on considering the supremal 
(A, B)-invariant subspaces contained in the kernels of the various subsystems formed by deleting 
the outputs corresponding to each desired block in turn. The ranges of these subspaces deter- 
mine whether decoupling is possible by state feedback. If it is not, the related “efficient extension” 
approach allows a precompensator of relatively low order to be found that will produce the desired 
block diagonal structure. This approach is somewhat involved, and the interested reader is referred 
to Morse and Wonham [8] for further details. 


27.1.5 Triangular Decoupling 


There is a form of “partially decoupled” system that can be of particular value for certain plants. This is 
the triangularized form, where all entries of the closed-loop transfer matrix above its leading diagonal 
are made zero. The first closed-loop output, y;, is therefore affected only by the first input 7; the second, 
y2, is influenced only by inputs r; and 1; etc. This type of transfer matrix can be used in the following 
sequential control scheme. First, input r; is adjusted until output y is as desired, and the control is then 
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frozen. Output variable y2 is then affected only by rz and the fixed 1), so rz can be adjusted until this 
output is also as desired. The third input, r3, can then be used to set output y3, etc. This scheme can be 
seen to be less powerful than diagonal decoupling, as the outputs must be adjusted sequentially rather 
than fully independently. However, it has one strong point in its favor: any right-invertible plant can be 
triangularized by state feedback alone, regardless of whether additional precompensation is required to 
make it diagonally decoupled. Proof of this follows directly from the fact that there always exists some 
state feedback gains F, G for which Tp,g(s) = X; 1(s), and the interactor is, by definition, lower triangular. 
Of course, similar results apply for generalized rather than standard interactors also. Therefore, it can 
be shown, as originally proved by Morse and Wonham [9], that all closed-loop poles of the triangularly 
decoupled system can be arbitrarily assigned. 

Finally, it may also be possible to triangularize a system by means of the simpler constant output 
feedback. If the original plant is square and strictly proper (D = 0), it can be shown that this is possible if 
and only if all entries of the modified inverse transfer matrix Ky T~ !(s) that lie above the leading diagonal 
are constant. This is quite a simple condition to test and is very similar to the test for diagonal decoupling 
by output feedback. The required gain matrix H is given from the fact that the upper triangular part of 
KrH is precisely the upper triangular constant part of Kr T~!(s). It can be noted that there is therefore 
some non-uniqueness in the choice of the gain H: in particular, we can add a term of the form K; 'Z 
to H, where Z is any lower triangular constant matrix, and still get a suitable output gain matrix. If it is 
possible to triangularize a given system by output feedback, there is consequently some freedom to assign 
closed-loop poles also. However, it is difficult to quantify this freedom in any concrete way. 


27.1.6 Static Decoupling 


Static decoupling, as opposed to dynamic decoupling already described, is much easier to achieve. A 
system is statically decoupled if a step change in the (static) steady-state level of the ith input is reflected 
by a change in the steady-state level of the ith output and only that output. To derive the conditions for 
static decoupling, assume that the system is described by a p x p transfer matrix T(s) that is bounded- 
input/bounded-output stable; that is, all of its poles are in the open left half of the s-plane and none is 
on the imaginary axis. Note that stability is necessary for the steady-state values of the outputs to be 
well defined. Assume now that the p inputs are step functions described by u;(s) = s i=1,...,p. The 
steady-state value of the output vector y, y,5, can then be found using the final value theorem, as follows: 


ky ky 
Lik ky 
¥ss = lim y(t) = lim sT(s)— =T(0)] . (27.10) 
S—>0O s>0 Ss : : 
kp kp 


It is now clear that T(s) is statically decoupled if and only if T(0) is a diagonal nonsingular matrix; 
that is, all the off-diagonal entries of T(s) must be divisible by s, while the entries on the diagonal should 
not be divisible by s. It can be shown easily that a system described by a p x p transfer matrix T(s) that is 
bounded-input/bounded-output stable can be statically decoupled, via u = Gr, if and only if 


rank T(0) =p (27.11) 


that is, if and only if there is no transmission zero at s = 0. Note that this condition, if a controllable and 
observable state-space description is given, is 


A B 
rank e p)=nte (27.12) 
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If this is the case, any feedforward constant gain G, in u= Gr, such that T(0)G is a diagonal and 
nonsingular matrix will statically decouple the system. To illustrate, consider the following example: 


Example 27.9: 
s+2 2 
_ | s+ s+3 
TO =] saa) 1 
(s+3)2 s+] 
2 2/3 ; 
Here T(0) = 0 1): which has full rank; therefore, it can be statically decoupled. Let T(0)G = 
2 0). _fl =1f 
ic 1) ithen = (9 1 ) Note that 
s+2 —s(s — 1) 
s+] 3(s + 1)(s+ 3) 
T(s)G = 3 2 
s(s+1) —s?+s*+175+27 


(s+3)2 3(s + 1)(s +3)? 
where all the off-diagonal entries of T(s) are divisible by s, while the entries on the diagonal are not 


divisible by s. If now the input 7 @ is applied to T(s)G, the steady-state output is T(0)G @ = 
2 2 
2k; 
kp } 


27.2 Defining Terms 


Decoupling: Separating the system into a number of independent subsystems. 

Non-interacting control: The control inputs and the outputs can be partitioned into disjoint subsets; 
each subset of outputs is controlled by only one subset of inputs, and each subset of inputs 
affects only one subset of outputs. From an input/output viewpoint, the system is split into 
independent subsystems; it is called decoupled. 
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Model predictive control (MPC) refers to a class of control algorithms that compute a sequence of control 


moves based on an explicit prediction of outputs within some future horizon. The computed control 


moves are typically implemented in a receding horizon fashion, meaning only the moves for the current 


time are implemented and the whole calculation is repeated at the next sample time. In essence, MPC is 
a feedback control strategy based on repeated calculation of open-loop control trajectories. 

It is difficult to attribute MPC to any single individual, since the idea of MPC has appeared in many 
different forms and in the context ofa variety of applications. In the process industries, serious applications 


and research on the subject began in the late 1970s, fueled by seminal papers by several industrialists, 


who outlined the basic algorithm and pointed out their potential for providing effective solutions to 
difficult process control problems [1,6]. Owing to its unique ability to handle process interactions and 
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constraints in a unified manner, MPC progressed rapidly, establishing an impressive track record along 
the way. The initial applications were mainly in the petrochemical industries, but it has been applied to 
a variety of industries including chemicals, food, and pulp and paper. Now MPC has become a standard 
tool for process control and there are several vendors that market general-purpose MPC software and 
commissioning services. 

The objective of this chapter is to introduce the linear MPC technique as viewed from the process 
industries. After introducing a prototypical algorithm in Section 28.2, we discuss some implementation 
issues in Section 28.3 and introduce several notable idiosyncratic features of various other commercial 
MPC algorithms. Finally, we point out some future research needs. 


28.2 Industrial MPC Algorithm 


Dynamic matrix control (DMC) was one of the first commercial implementations of MPC. In this section, 
we describe the basic ideas of the algorithm. 


28.2.1 The Idea of Moving Horizon Control 


Consider the diagram in Figure 28.1. At the present time k the behavior of the process over a horizon p is 
considered. Using the model, the response of the process output to changes in the manipulated variable 
(MV) is predicted. Current and future moves of the MVs are selected such that the predicted response has 
certain desirable (or optimal) characteristics. For instance, a commonly used objective is to minimize the 
sum of squares of the future errors, that is, the deviations of the controlled variable (CV) from a desired 
target (setpoint). This minimization can also take into account constraints, that may be present on the 
MVs and the outputs. 

The idea is appealing but would not work very well in practice if the moves of the MV determined 
at time k were applied blindly over the future horizon. Disturbances and modeling errors may lead to 
deviations between the predicted behavior and the actual observed behavior, so that the computed MV 
moves may not be appropriate any more. Therefore only the first one of the computed moves is actually 
implemented. At the next time step k + 1, a measurement is taken, the horizon is shifted forward by one 


Past Future 


Manipulated u(k + i) 
inputs 


tk+1 k+p 


kt lk+1 kp 


FIGURE 28.1 Moving horizon control. 
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step, and the optimization is done again over this shifted horizon based on the updated prediction of the 
system behavior. Therefore, this control strategy is also referred to as moving horizon control. 

A similar strategy is used in many other nontechnical situations. One example is computer chess where 
the computer moves after evaluating all possible moves over a specified “depth” (the horizon). At the 
next turn the evaluation is repeated based on the current board situation. Another example would be 
investment planning. A five-year plan is established to maximize the return. Periodically, a new five-year 
plan is put together over a shifted horizon to take into account changes that have occurred in the economy. 

The DMC algorithm includes as one of its major components a technique to predict the future output 
of the system as a function of the inputs and disturbances. This prediction capability is necessary to 
determine the optimal future control inputs and will be discussed next. Afterward we will state the 
objective function, and formulate the optimization problem. 


28.2.2 Multistep Prediction 


We consider the setup depicted in Figure 28.2 where we have three different types of external inputs: the 
MV u, whose effect on the output, usually a CV, is described by P,,; the measured disturbance variable 
(DV) d whose effect on the output is described by Pg; and finally the unmeasured and unmodeled 
disturbances wy, which add a bias to the system output. The overall system can be described by 


y(k) =[P*  P4] | + wy(k). (28.1) 


We assume that step response models S“, S@ are available for the system dynamics P,, and Py, respec- 
tively. We can define the overall multivariable step response model 


S=[S* 5": (28.2) 
which is driven by the known overall input 
__ | Au(k) 
Av(k) = eal : (28.3) 


Let us adopt as the system state 


VwH=[7H, FT, .... Fw)’, (28.4) 


where nis the number of sample steps it takes for the system to settle down after a step change is made to 
any of the inputs. The elements of the state represent the future system outputs 


yk) 
y(k +1) 


=< 

~ 

a 
I 


(28.5) 
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FIGURE 28.2 Basic problem setup. 
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obtained under the assumption that the system inputs do not change from the previous values, that is, 


Au(k) = Au(k +1) =---=0, 
Ad(k) = Ad(k+1)=---=0. 


(28.6) 


Also, the state does not include any unmeasureed disturbance information and hence it is assumed in 
the definition that 


wy(k) = wy(k+1)=---=0. (28.7) 
The state is updated according to 
Y¥(k) =M-Y(k—1)+ SAv(k— 1). (28.8) 


where M is a shift operator expressed as 


0 I Oo see dnca~ (OQ: 20 
0 0 TI 0... 0 0 
M=|: n (28.9) 
0 0 01 
0 0 0 oT 
and 
Sj 
SS) ly (28.10) 
Sn 


where S; is the ith step response coefficient matrix. The equation reflects the effect of the input change 
Av(k— 1) on the future evolution of the system assuming that there are no further input changes. The 
influence of the input change manifests itself through the step response matrix S. The effect of any future 
input changes is described as well by the appropriate step response matrix. Let us consider the predicted 
output over the next p time steps 


y(k + 1k) yilk) St 0 
wk+2IK) | | jal] |S} si 
=) ol) ep Ankiae| 2 | Autee aes pes 
yk+plk)| Lip] LS? Sp-1 
0 s 0 
0 S} s 
+]: | Auk+p-Ukb+| : | Ad& t+] % | Adk+1lk)+--- 
0 
st si Spt 
0 wy(k + 1|k) 
0 wy(k + 2|k) 
+]: | Ad(kt+p—1k)+ : (28.11) 
0 
sf 


Here the first term on the right-hand side, the first p elements of the state, describes the future evolution 
of the system when all the future input changes are zero. The remaining terms describe the effect of the 
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present and future changes of the manipulated inputs Au(k + i|k), the measured disturbances Ad(k + i|k), 
and the unmeasured and unmodeled disturbances w,(k + i|k). The notation y(k + ilk) represents the 
prediction of y(k +i) made based on the information available at time k. The same notation applies to 
Ad and wy. 

The values of most of these variables are not available at time k and have to be predicted in a rational 
fashion. From the measurement at time k d(k) is known and therefore Ad(k) = d(k) — d(k — 1). Unless 
some additional process information or “upstream” measurements are available to conclude about the 
future disturbance behavior, the disturbances are assumed not to change in the future for the derivation 
of the DMC algorithm. 


Ad(k + 1|k) = Ad(k + 2|k) =--» = Ad(k-+p—1|k) =0. (28.12) 


This assumption is reasonable when the disturbances are varying only infrequently. Similarly, we will 
assume that the future unmodeled disturbances wy(k + i|k) do not change. 


wy(k|k) = wy(k + 1|k) = wy(k + 2|k) = +++ = wy(k + pik). (28.13) 
We can obtain an estimate of the present unmodeled disturbance from Equation 28.1 
wy(klk) © ¥m(k) — yolk), (28.14) 


where ym(k) represents the value of the output as actually measured in the plant. Here yo(k), the first 
component of the state Y(k), is the model prediction of the output at time k (assuming wy(k) = 0) based 
on the information up to this time. The difference between this predicted output and the measurement 
provides a good estimate of the unmodeled disturbance. 

For generality, we want to consider the case where the manipulated inputs are not varied over the 
whole horizon p but only over the next m steps (Au(k|k), Au(k+ 1|k),..., Au(k-+ m— 1|k)) and that 
the input changes are set to zero after that. 


Au(k-+ mk) = Au(k-+m-+ Ik) = ++» = Au(k+p—1)|k) =0. (28.15) 


With these assumptions Equation 28.11 becomes 


jr(k) sf ymlk) — Folk) 
y2(k) S Ym(k) — Yolk) 
Wk + 1k) = +]: | Ad(k) + 
p(k) SS y¥m(k) — Folk) 
—S—S/ >_> ——<———S —<$<—-_ -—_— 
MY (k) S4Ad(k) Lp(ym(k) — yo(k)) 
from the memory feedforward term feedback term 
u 0 oh hes 0 
oe aoe. Au(kik) 
2 : Au(k + 1k) 
+ sh st s (28.16) 
u ‘si u Au(k +m — 1k) 
Se Ss awed 


a AU(k) 
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Here we have introduced the new symbols 


Wk + 1k) 
yk + 2|k) 
V(k+ 1k) = : ; (28.17) 
y(k + plk) 
sé 0 0 
SOS} 0 sd 
5 1 |S 
cin ae ee ae Als (28.18) 
‘ d 
: S5 
Sp Spy So—m-+1 
i 
I 
Lp = p> (28.19) 
I 
Au(k\k) 
Au(k + 1k) 
AU (k) = . , (28.20) 
Au(k +m — 1k) 
0 I 0 0 
0 0 I 0 0 
k, ; p forp<n, 
0 0 I 0 0 
eI OI 0 he8 (28.21) 
0 0 I 0 
. p forp>n, 
0 0 I 
Oy S54 lee, OT 
With this new notation the p-step ahead prediction becomes 
V(k + 1k) = MY(k) + S4Ad(k) + Lp(Ym(k) — yo(k)) + S“ AU (k), (28.22) 


where the first three terms are completely defined by past control actions (Y(k), jo(k)) and present 
measurements (y(k), Ad(k)) and the last term describes the effect of future MV moves AU/(k). 

This prediction equation can be easily adjusted if different assumptions are made on the future behavior 
of the measured and unmeasured disturbances. For instance, if the disturbances are expected to evolve in 
a ramp-like fashion then we would set 


Ad(k) = Ad(k + 1k) =+-» = Ad(k +p — 1k) (28.23) 
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and 
Wy(k + £|k) = wy(k]k) + €(wy (klk) — wy(k — 1k — 1). (28.24) 


28.2.3 State-Space Formulation 


Although the first generation of industrial algorithms adopted a finite impulse response or a step response 
model, given their intuitive appeal to the practitioners, its limitations were quickly pointed out by the 
academics who went on to propose more general formulations based on state-space models. These formu- 
lations were later adopted by the vendor companies to develop a second generation of commercial MPC 
algorithms. These formulations also incorporated state estimation techniques into MPC for enhanced 
disturbance estimation and noise filtering. 

The state-space model form used was 


x(k + 1) = Ax(k) + B,u(k) + Byw(k) + Bgd(k), 
y(k) = Cx(k) + wy(k). 


Here wy and Wy are state and output disturbances. Since these disturbances tend to be “persistent," 
integrating states are created to reflect their nature: 


Wx(k + 1) = wx(k) + €1(k); 


(28.25) 


28.26 
wy(k+ 1) = wy(k) +e2(b), oe 
where €; and €2 are white-noise sequences. The above can be expressed as 
x(k +1) A By, O x(k) By Bg 0 0 e1(k) 
wektD}=]0 TF O}] luk) | +] 0]ub+}]0]aq&+]r 0 Bae 
wy(k + 1) 0 0 IJ Lwy(k) 0 0 (a ae 
(28.27) 
x 
y(k)=[C 0 I]| wx(k) ], 
wy (k) 
which will be denoted using the notation 
k+1) = Oz(k) + Tyu(k) + Pady + Pee(k), 
2(k + 1) = &z(k) u(k) + Pad, + Vee(k) (28.28) 


y(k) = Bz(k). 


The above is a standard state-space model with white state noise input and Kalman filtering can be 
used to design a state estimator of the following form: 


2(k|k — 1) = @3(k— 1k — 1) +Tuulk) +P ad, 
2(k|k) = 2(k — 1k —1) + K (p(k) — B2(kKlk— 1), 


(28.29) 


where K is the Kalman filter gain. 

One potential problem for using the above model is the lack of detectability: One cannot have more dis- 
turbance states (wy and wy), which have integrating dynamics, than the number of outputs for detectability 
to hold. Hence, to use the above model, one would have to choose between input and output disturbances. 
Alternatively, it may be convenient to adopt the following differenced model form: 


Ax(k+1)]_ [A 0] [x(k] , [ Bu By By 0] fex(k) 
| y(k +1) a B A il bs Fal BuO el AG ke | ke 


Ax(k 
y(k)=[0 I] ea 


The above system is detectable as long as the original (C, A) was a detectable pair. Once again, the above 
system is in the standard state-space form of Equation 28.28 and the Kalman filter of Equation 28.29 can 
be designed for state estimation. 


(28.30) 
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The multistep prediction equation can be constructed and has the following structure: 
Vk + 1k) = S22(klk) + S4Ad(k) + S“AU(k). (28.31) 


The readers are referred to [4] for a detailed derivation of the matrices in the above equation. The above 
equation has the same linear structure of V(k + 1|k) = b(k) + S“ AU(k) as before and the subsequent 
derivations presented hereafter hold for both the step response model-based formulation and the 
state-space model-based formulation (with some obvious modifications). 


28.2.4 Objective Function 


Plant operation requirements determine the performance criteria of the control system. These criteria 
must be expressed in mathematical terms so that a control law can be obtained in algorithmic form. In 
DMC, a quadratic objective function is used, which can be stated in its simplest form as* 


P 
F 2 

ee eee d lyk + elk) — r(k + O° (28.32) 
This criterion minimizes the sum of squared deviations of the predicted CV values from a time-varying 
reference trajectory or setpoint r(k + @) over p future time steps. The quadratic criterion penalizes large 
deviations proportionally more than smaller ones so that on the average the output remains close to its 

reference trajectory and large excursions are avoided. 
Note that the MVs are assumed to be constant after m intervals of time into the future, or equivalently, 


Au(k+ m|k) = Au(k+m-+ 1|k) =--- = Au(k+ p— 1|k) =0, 


where m < p always. This means that DMC determines the next m moves, only. The choices of m and 
p affect the closedly-loop behavior. Moreover, m, the number of degrees of freedom, has a dominant 
influence on the computational effort. Also, it does not make sense to make the horizon longer than 
m+n (p<m-+n), because for an FIR system of order n the system reaches a steady state after m+n 
steps. Increasing the horizon beyond m+n would simply add identical constant terms to the objective 
function (Equation 28.32). 

Due to inherent process interactions, it is generally not possible to keep all outputs close to their 
corresponding reference trajectories simultaneously. Therefore, in practice only a subset of the outputs 
is controlled well at the expense of larger excursions in others. This can be influenced transparently by 
including weights in the objective function as follows: 


P 
min Pyke + elk) — r(k + 8)]|\7 28.33 
ab eso Lyk + fk) — rk + OI (28.33) 


For example, for a system with two outputs y; and y2, and constant diagonal weight matrices of the form 
0 
P= i » VE (28.34) 


the objective becomes 


P P 
vv? Silk + 2k) — n(k+ OF +2? So balk+ alk) — nie+or| . (28.35) 


min | 
Au(k\k)... Au(k+m—1|k 
u(k|k)...Au(k+m—1|k) ram FA 


Thus, the larger the weight is for a particular output, the larger is the contribution of its sum of squared 
deviations to the objective. This will make the controller bring the corresponding output closer to its 
reference trajectory. 


* — \|x|| denotes the norm (x7 x)1/2 of the vector x. 
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Finally, the MV moves that make the output follow a given trajectory could be too severe to be 
acceptable in practice. This can be corrected by adding a penalty term for the MV moves to the objective 
as follows: 


P m 
- YoU k+ lk) — rk + OU? + DS WN eLAu(k + é— DIP. (28.36) 
l=1 l=1 


Note that the larger the elements of the matrix Ij, the smaller the resulting moves, and consequently, 
the output trajectories will not be followed as closely. Thus, the relative magnitudes of I” fj and I will 
determine the trade-off between following the trajectory closely and reducing the action of the MVs. 

Of course, not every practical performance criterion is faithfully represented by this quadratic objective. 
However, many control problems can be formulated as trajectory tracking problems and therefore this 
formulation is very useful. Most importantly this formulation leads to an optimization problem for which 
there exist effective solution techniques. 


28.2.5 Constraints 


In many control applications the desired performance cannot be expressed solely as a trajectory following 
the problem. Many practical requirements are more naturally expressed as constraints on process vari- 
ables. 

There are three types of process constraints: 


MV constraints: these are hard limits on inputs u(k) to take care of, for example, valve saturation con- 


straints. 
MV rate constraints: these are hard limits on the size of the MV moves Au(k) to directly influence the 
rate of change of the MVs. 


Output variable constraints: hard or soft limits on the outputs of the system are imposed to, for example, 
avoid overshoots and undershoots. These can be of two kinds: 
« CY: limits for these variables are specified even though deviations from their setpoints are 
minimized in the objective function 
« Associated variables: no setpoints exist for these output variables but they must be kept within 
bounds (i.e., corresponding rows of I" . are zero for the projections of these variables in the 
objective function given in Equation 28.36. 


The three types of constraints in DMC are enforced by formulating them as linear inequalities. In the 
following, we explicitly formulate these inequalities. 


28.2.5.1 MV Constraints 


The solution vector of DMC contains not only the current moves to be implemented but also the moves 
for the future m intervals of time. Although violations can be avoided by constraining only the move 
to be implemented, constraints on future moves can be used to allow the algorithm to anticipate and 
prevent future violations, thus producing a better overall response. The MV value at a future time k + @ 
is constrained to be 


L 
Ujow(l) <)> Au(k + jk) + ulk—1) < unign(Qs &=0,1,...m—1, 
j=0 


where u(k—1) is the implemented previous value of the MV. For generality, we allowed the limits 
Ulow(£), Unign(€) to vary over the horizon. These constraints are expressed in matrix form for all 
projections as 
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u(k — 1) — upign(0) 
-h u(k — 1) ~ upigam — 1) 
ie auee | ote oe 


Ulow(m ~ 1) = u(k = 1) 


where 
IO 0 
I I «-. 0 
h=|. . . . |. (28.38) 
It I 


28.2.5.2 MV Rate Constraints 


Often MPC is used in a supervisory mode where there are limitations on the rate at which lower-level 
controller setpoints are moved. These are enforced by adding constraints on the MV move sizes: 


—Aumax(0) 


—I ee eGa ee 1) 
| ; HACE) 3) A ey (28.39) 


—AUmax(m = 1) 
where Atmax(€) > 0 is the possibly time-varying bound on the magnitude of the moves. 


28.2.5.3 Output Variable Constraints 


The algorithm can make use of the output predictions (Equation 28.22 or 28.31) to anticipate future 
constraint violations: 


Viow < Vk + 1k) < Vnigh- (28.40) 


Substituting from Equation 28.22, we obtain constraints on AU/(k): 


oc MY (K) + S4Ad(K) + Zp(n() — Fo(K)) — Vr 
| Ss" ate neo —jo(k))) + Yiow | (28.41) 
where 
Ylow(1) Yhigh (1) 
Yiow(2) Ynigh(2) 
low = . > Vhigh = : 
low (P) high (P) 


are vectors of output constraint trajectories yio,,(2), Yhigh (2) over the horizon length p. 
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28.2.5.4 Combined Constraints 


The MV constraints (Equation 28.37), MV rate constraints (Equation 28.39) and output variable con- 
straints (Equation 28.41) can be combined into one convenient expression 


C“AU(k) > C(k + 1k), (28.42) 
where C” combines all the matrices on the left-hand side of the inequalities as follows: 


a 
I, 
aie tte 
C= 1 |: (28.43) 
su 
&" 
The vector C(k + 1|k) on the right-hand side collects all the “error” vectors on the constraint equations 


as follows: 
u(k — 1) — unign(0) 


u(k — 1) — pig (on — 1) 
Ujow(0) = u(k — 1) 


eG Dae 


— Atimax(0) 
C(k + 1|k) = (28.44) 
—Atimax(m = 1) 
— Atimax(0) 
—Autmax(m ~ 1) 
MY (k) + S4Ad(k) + Zp(¥m(k) — Folk) — Vnign 
—(MY(k) + S4Ad(k) + Zp(¥m(k) — Fo(k))) + Viow 
28.2.6 Formulation of Control Problem as a Quadratic Program 
We make use of the prediction equation 28.22 to rewrite the objective 
P m 
pain. Yo Ik + alk) = rk + OUP + D> INE LAw(k + 6— DIP (28.45) 
é=1 é=1 


and add the constraints (Equation 28.42) to obtain the optimization problem 
gain {IID [VK + 1k) — RF YIP + AUDI} 


st. Y(k+ 1k) = MY (k) + S4Ad(k) + Zp(ym(k) — jo(k)) + SYAU(K)CYAU(k) = Clk + 114), 
(28.46) 


where 
Pe digg | iss sal get (28.47) 
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and 

TY =diag {Iy, ---,Tp} (28.48) 
are the weight matrices in block diagonal form, and 


r(k +1) 


r(k +2) 
R(k+1)= ; (28.49) 


r(k +p) 


is the vector of reference trajectories. 
We can substitute the prediction equation into the objective function to obtain 


WY [VR + 1k) — R(k+ 1] II? + IP“ AU) |I? (28.50) 
= ||P" [S“AU(k) — Ep(k + 1k)] |? + IT“ AUK) II (28.51) 
= AUT (k)(SY¥ T/T TP’ Ss” + F¥ PT) AU(k) (28.52) 


— 2Ep(k + 1k)" PPS" AU(k) + ES (k+ Uk)DY PE, (k + 1k). 


Here we have defined 


e(k +1]k) 
(k-+ 21k) 

Ep(k + 1|k) = (28.53) 
e(k + pik) 


ER(k+1)— [mM ¥ (k) + S*Ad(k) + Zp(ym(k) — o(h))] : 


which is the measurement corrected vector of future output deviations from the reference trajectory 
(i.e., errors), assuming that all future control moves are zero. Note that this vector includes the effect of 
the measurable disturbances (S4 Ad(k)) on the prediction. For the state-space model of Equation 28.31, Ep 
changes to 


Bp 2 Rk + 1) — (S*2(kIk) + S4Ad(h)).. (28.54) 


The optimization problem with a quadratic objective and linear inequalities, which we have defined, is a 
quadratic program (QP). By converting to the standard QP formulation the DMC problem becomes* 


1 
min —AU(k)'H"“AU(k) — Glk + 1k)? AU(k) 
AU(k) 2 

st. C“AU(k) > C(k + 1k), (28.55) 


where the Hessian of the QP is 


HY = stp tps 4 prety (28.56) 


* The term Ep (k + 1k)Ep(k + 1|k) is independent of A?/(k) and can be removed from the objective function. 
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and the gradient vector is 


G(k+ 1k) = S"" TYP Ep(k + Ik). (28.57) 


28.3 Implementation Issues 


As explained in the introduction of this chapter the implementation of DMC is done in a moving horizon 
fashion. This implies that the QP derived above will be solved at each controller execution time. Because 
of this feature, the algorithm can be configured online as required to take care of unexpected situations. 
For example, in case an actuator is lost during the implementation, the high and low constraint limits 
on that particular MV can be set to be equal. Then the MPC problem with the remaining MVs is solved. 
Similarly, the weight parameters in the objective function can also be adjusted online, giving the user the 
ability to tune the control law. In this section we discuss the different implementation issues associated 
with the DMC. 


28.3.1 Moving Horizon Algorithm 


The constrained MPC algorithm is implemented online as follows. 


1. Preparation. Do not vary the MVs for at least n time intervals (Au(—1) = Au(—2)=---= 
Au(—n) =0) and assume the measured disturbances are zero (Ad(—1) = Ad(—2) =---= 
Ad(—n) = 0) during that time. Then the system will be at rest at k = 0. 

2. Initialization (k = 0). Measure the output y,,(0) and initialize the model prediction vector as* 


T 
¥(k) = | ym(0)"s¥m(0)">---2%m(0)? | - (28.58) 
————$— eee 
3. State update: Set k = k+ 1. Then, update the state according to 
Y(k) =M- ¥(k—1)+S“Au(k—1) + S4Ad(k— 1), (28.59) 


where the first element of Y(k), (kk), is the model prediction of the output y,,(k) at time k. 
4. Obtain measurements: Obtain measurements (y(k), Ad(k)). 
5. Compute the reference trajectory error vector 


E(k + 1k) = R(k +1) — MY (k) + S4Ad(k) + Lp(ym(k) — yoCk)). (28.60) 
6. Compute the QP gradient vector 


Gk + 1|k) = S"7 (1) PE, (k + 1k). (28.61) 


If Equation 28.58 is used for intialization and changes in the past n inputs did actually occur, then the initial operation 
of the algorithm will not be smooth. The transfer from manual to automatic will introduce a disturbance; it will not be 
“bumpless.” 
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7. Compute the constraint equation’s right-hand side vector 
u(k — 1) = upign(0) 


u(k — 1) — unign(m — 1) 
Ulow (0) - u(k 7; 1) 


pin eed 


—Aumax(0) 
C(k+ 1|k) = : 


(28.62) 
aes _ 1) 
—Aumax (0) 


—AUmax(m _ 1) 
—Ep(k + 1k) + R(k+ 1) — Vnigh 
Ep(k + 1k) —RK+1) + Yiow 


8. Solve the QP 


minay(k 7AU(K)TH"AU(Kk) — G(k + 1k)? AU(k) (28.63) 

s.t. CYAU(k) = C(k+ 1k) : 
and implement Au(k|k) as Au(k) on the plant. 

9. Goto 3. 


Note that the sequence of moves produced by the moving horizon implementation of the QP will be 
different from the sequence of moves AU/(k). 


28.3.2 Solving the QP 


In a moving horizon framework the QP in Equation 28.63 is solved at each controller execution time after 
a new prediction is obtained. The only time-varying elements in this problem are the vectors E,(k + 1|k) 
(or equivalently G(k + 1|k)) and C(k+ 1|k). That is, the Hessian H” of the QP remains constant for all 
executions. In that case, as explained above, a parametric QP algorithm which employs the preinverted 
Hessian in its computations, is preferable in order to reduce online computation effort. Of course, in 
case either I’ or I“ (or the step response coefficients) need to be updated, or the model’s step response 
coefficients have changed, the Hessian must be recomputed and inverted in background mode in order 
not to increase the online computational requirements. 

QP is a convex program and therefore, is fundamentally tractable, meaning a global optimal solution 
within a specified tolerance can be assured. Although not as extensively as linear programs (LPs), QPs 
have been well studied and reliable algorithms have been developed and coded. General-purpose QP 
solvers like QPSOL are readily available but use of tailored algorithms that take advantage of specific 
problem structures can offer significant computational savings. 

The conventional approach for solving QPs is the so-called active set method. In this method, one 
initiates the search by assuming a set of active constraints. For an assumed active set, one can easily 
solve the resulting least-squares problem (where the active constraints are treated as equality constraints) 
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through the use of Lagrange multiplier. In general, in the active set one starts out with that it will not be 
the correct one. Through the use of the Karush-Kuhn-Tucker (KKT) condition,* one can modify the 
active set iteratively until the correction is found. Most active set algorithms are feasible path algorithms, 
in which the constraints must be met at all times. Hence, the number of constraints can have a significant 
effect on the computational time. 

More recently, a promising new approach called the interior point (IP) method has been getting a lot 
of attention. The idea of the IP method is to “trap” the solution within the feasible region by including a 
so-called “barrier” function in the objective function. With the modified objective function, the Newton 
iteration is applied to find the solution. Though originally developed for LPs, the IP method can be 
readily generalized to QPs and other more general constrained optimization problems. Even though not 
formally proven, it has been observed empirically that the Newton iteration converges within 5-50 steps. 
Significant work has been carried out in using this solution approach for solving QPs that arise in MPC, 
but details are out of the scope of this chapter. 

Computational properties of QPs vary with problems. As the number of constraints increases, more 
iterations are generally required to find the QP solution, and therefore the solution time increases. This 
may have an impact on the minimum control execution time possible. Also, note that the dimension of 
the QP (i.e., the number of degrees of freedom m - n,) influences the execution time proportionately. 

Storage requirements are also affected directly by the number of degrees of freedom and the number 
of projections n- ny. For example, the Hessian size increases quadratically with the number of degrees 
of freedom. Also, because of the prediction algorithm, Y(k) must be stored for use in the next controller 
execution (both E,(k + 1|k) and C(k + 1|k) can be computed from Y(k)). 


28.3.3 Proper Constraint Formulation 


Many engineering control objectives are stated in the form of constraints. Therefore, it is very tempting 
to translate them into linear inequalities and to include them in the QP control problem formulation. In 
this section we want to demonstrate that constraints make it very difficult to predict the behavior of the 
control algorithm under real operating conditions. Therefore, they should be used only when necessary 
and then only with great caution. 

First of all constraints tend to greatly increase the time needed to solve the QP. Thus, we should 
introduce them sparingly. For example, if we wish an output constraint to be satisfied over the whole 
future horizon, we may want to state it as a linear inequality only at selected future sampling times rather 
than at all future sampling times. Unless we are dealing with a highly oscillatory system, a few output 
constraints at the beginning and one at the end of the horizon should keep the output more or less inside 
the constraints throughout the horizon. Note that even when constraint violations occur in the prediction, 
this does not imply constraint violations in the actual implementation because of the moving horizon 
policy. The future constraints serve only to prevent the present control move from being short-sighted. 

Output constraints can also lead to an “infeasibility.” A QP is infeasible if there does not exist any value 
of the vector of independent variables (the future MV move Al/(k)), which satisfies all the constraints — 
regardless of the value of the objective function. Physically, this situation can arise when there are output 
constraints to be met but the MVs are not sufficiently effective—either because they are constrained or 
because there is deadtime in the system that delays their effect. Needless to say, provisions must be built 
into the online algorithm such that an infeasibility never occurs. 

Mathematically an infeasibility can only occur when the right-hand side of the output constraint 
equations is positive. This implies that a nonzero move must be made in order to satisfy the constraint 
equations. Otherwise, infeasibility is not an issue since AU/(k) = 0 is feasible. 

A simple example of infeasibility arises in the case of deadtimes in the response. For illustration, assume 
a single-input single-output (SISO) system with 8 units of deadtime. The output constraint equations for 


* The KKT condition is a necessary condition for the solution to a general constrained optimization problem. For QP, it is 


a necessary and sufficient condition. 
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this system will look like 


0 0 0 c(k + 1|k) 

0 0 0 c(k + 8|k) 
-St 0 g| AM | a4 11K) 
0 c(k+0+2|k) 


u u 
—So42 641 


Positive elements c(k + 1|k),...,c(k-+6|k) indicate that a violation is projected unless the MVs are 
changed (AU/(k) 4 0). Since the corresponding coefficients in the left-hand side matrix are zero, the 
inequalities cannot be satisfied and the QP is infeasible. Of course, this problem can be removed by 
simply not including these initial @ inequalities in the QP. 

Because inequalities are dealt with exactly by the QP, the corrective action against a projected violation 
is equivalent to that generated by a very tightly tuned controller. As a result, the moves produced by the 
QP to correct for violations may be undesirably severe (even when feasible). Both infeasibilities and severe 
moves can be dealt with in various ways. 

One way is to include a constraint window on the output constraints similar to what we suggested 
above for computational savings. For each output a time k + H, in the future is chosen at which constraint 
violations will start to be checked (Figure 28.3). 

For the above illustration, this time should be picked to be at least equal to 8+ 1. This allows the 
algorithm to check for violations after the effects of deadtimes and inverse responses have passed. For 
each situation there is a minimal value of H, necessary for feasibility. If this minimal value is chosen large, 
constraint violations may occur over a significant period of time. In many cases, if a larger value of H, 
is chosen, smaller constraint violations may occur over a longer time interval. Thus, there is a trade-off 
between magnitude and duration of constraint violation. 

In general, it is difficult to select a value of H, for each constrained output such that the proper com- 
promise is achieved. Furthermore, in multivariable cases, constraints may need to be relaxed according 
to the priorities of the constrained variables. The selection of constraint windows is greatly complicated 
by the fact that appropriate amount and location for relaxation are usually time dependent due to varying 
disturbances and occurrences of actuator and sensor failures. Therefore, it is usually preferred to “soften" 
the constraint by adding a slack variable € and penalizing this violation through an additional term in the 
objective function. 


min [Usual Objective] + ne? 
€, AU (k) 


Ymin — € < y(k+ £|k) < Vmax + € 


plus other constraints. 


I 
| 
ptt 
k k+H, 
Sa 
Relax the constraints between 
k+landk+H1 


Vmax 


FIGURE 28.3 Relaxing the constraints. 
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The optimization seeks a compromise between minimizing the original performance objective and 
minimizing the constraint violations expressed by ¢”. The parameter ) determines the relative importance 
of the two terms. The degree of constraint violation can be fine-tuned arbitrarily by introducing a separate 
slack variable € for each output and time step, and associating with it a separate penalty parameter }. 

Finally, we must realize that while unconstrained MPC is a form of linear feedback control, constrained 
MPC is a nonlinear control algorithm. Thus, its behavior for small deviations can be drastically different 
from that for large deviations. This may be surprising and undesirable and is usually very difficult to 
analyze a priori. 


28.3.4 Choice of Horizon Length 


On the one hand, the prediction horizon p and the control horizon m should be kept short to reduce 
the computational effort; on the other hand, they should be made long to prevent short-sighted control 
policies. Making m short is generally conservative because we are imposing constraints (forcing the 
control to be constant after m steps) that do not exist in the actual implementation because of the moving 
horizon policy. Therefore, a small m will tend to give rise to a cautious control action. 

Choosing p small is “short-sighted” and will generally lead to an aggressive control action. If constraint 
violations are checked only over a small control horizon p, this policy may lead the system into a “dead 
alley” from which it can escape only with difficulty, that is, only with large constraint violations and/or 
large MV moves. 

When p and m are infinity and when there are no disturbance changes and unknown inputs, the 
sequence of control moves determined at time k is the same sequence that is realized through the moving 
horizon policy. In this sense our control actions are truly optimal. When the horizon lengths are shortened, 
then the sequence of moves determined by the optimizer and the sequence of moves actually implemented 
on the system will become increasingly different. Thus the short-time objective, which is optimized, will 
have less and less to do with the actual value of the objective realized when the moving horizon control is 
implemented. This may be undesirable. 

In general, we should try to choose a small m to keep the computational effort manageable, but large 
enough to give us a sufficient number of degrees of freedom. We should choose p as large as possible, 
possibly oo, to completely capture the consequences of the control actions. This is possible in several ways. 
Because an FIR system will settle after m + n steps, choosing a horizon p = m + nisa sensible choice used 
in many commercial systems (Figure 28.4). 

Instead or in addition we can impose a large output penalty at the end of the prediction horizon forcing 
the system effectively to settle to zero at the end of the horizon. Then, with p = m+n, the error after 
m+ nis essentially zero and there is little difference between the finite and the infinite horizon objective. 


N time steps 


k+m-1 


FIGURE 28.4 Choosing the horizon. 
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28.3.5 Input Blocking 


As said, use of a large control horizon is generally preferred from the viewpoint of performance, but 
available computational resource may limit its size. One way to relax this limit is through a procedure 
called blocking, which allows to the user to “block out" the input moves at selected locations from the 
calculation by setting them to zero a priori. The result is a reduction in the number of input moves that 
need to be computed through the optimization, hopefully without a significant sacrifice in the solution 
quality. Obviously, judicious selection of blocking locations is critical for achieving the intended effect. 
The selection is done mostly on an ad hoc basis, although there are some qualitative rules like blocking 
less of the immediate moves and more of the distant ones. 
At a more general level, blocking can be expressed as follows: 


AU = BAU?, (28.64) 


where AU/? represents the reduced input parameters to be calculated through the optimization. B is the 
blocking matrix that needs to be designed for a good performance. Typically, the rows of B corresponding 
to the blocked moves would contain all zeros. In general, columns of B can be designed to represent 
different basis in the input space. Note that dimension of U/”, which is less than that of U/, must also be 
determined in the design. 


28.3.6 Filtering of the Feedback Signal 


In practice, feedback measurements can contain significant noise and other fast-varying disturbances. 
Since in DMC the effect of unmeasured disturbances is projected as a constant bias in the prediction, the 
high-frequency contents of a feedback signal must be filtered out in order to obtain a meaningful long- 
term prediction. For this, one can pass the feedback signal through a low-pass filter of some sort, perhaps 
a first- or second-order filter, before putting it into the prediction equation. Use of state estimation, 
discussed in [3], for example, allows one to model the statistical characteristics of disturbances and noise 
and perform the filtering in an optimal manner. 


28.4 Features Found in Other MPC Algorithms 


What we just covered is the basic form of a multivariable control algorithm called DMC, which was one 
of the first MPC algorithms applied to industrial processes with success. The original DMC algorithm did 
not use QP to handle constraints; instead, it added an extra output to the prediction to drive the input back 
to the feasible region whenever a predicted future input came close to a constraint. This was somewhat 
ad hoc and it was not until the 1980s that engineers at Shell Oil proposed the use of QP to handle input 
and output constraints explicitly and rigorously. They called this modified version QDMC. Currently, 
this basic form of DMC is still used in a commercial package called DMC-PLUS, which is marketed by 
Aspen Technology. 

Besides DMC and QDMC, there are several other MPC algorithms that have seen, and are still seeing, 
extensive use in practice. These include model predictive heuristic control (MPHC), which led to popular 
commercial algorithms such as IDCOM and SMC-IDCOM marketed by Setpoint (now Aspen Technol- 
ogy) and Hierarchical Constraint Control (HIECON) and Predictive Functional Control (PFC) marketed 
by Adersa; Predictive Control Technology (PCT), which was marketed by Profimatics (now Honeywell); 
and more recent Robust Model Predictive Control Technology (RMPCT), which is currently being mar- 
keted by Honeywell. These algorithms share same fundamentals but differ in details of implementation. 
Rather than elaborating on the details of each algorithm, we will touch upon some popular features not 
seen in the basic DMC/QDMC method. 
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28.4.1 Reference Trajectories 


In DMC, output deviation from the desired setpoint is penalized in the optimization. Other algorithms 
such as IDCOM, HIECON, and PFC let the user specify not only where the output should go but also 
how. For this, a reference trajectory is introduced for each CV, which is typically defined as a first-order 
path from the current output value to the desired setpoint. The time constant of the path can be adjusted 
according to the speed of the desired closed-loop response. This is displayed in Figure 28.5. 

Reference trajectories provide an intuitive way to control the aggressiveness of control, which is adjusted 
through the weighting matrix for the input move penalty term in DMC. One could argue that the 
controller’s aggressiveness is more conveniently tuned by specifying the speed of output response rather 
than through input weight parameters, whose effects on the speed of response is highly system dependent. 


28.4.2 Coincidence Points 


Some commercial algorithms such as IDCOM and PFC allowed the option of penalizing the output error 
only at a few chosen points in the prediction horizon called coincidence points. This is motivated primarily 
by reduction in computation it brings. When the number of input moves has to be kept small (in order 
to keep the computational burden low), use of a large prediction horizon, which is sometimes necessary 
due to large inverse responses, and long dynamics, results in a sluggish control behavior. This problem 
can be obviated by penalizing output deviation only at a few carefully selected points. At the extreme, one 
could ask the output to match the reference trajectory value at a single time point, which can be achieved 
with a single control move. Such formulation was used, for example, in IDCOM-M, an offspring of the 
original IDCOM algorithm, marketed by setpoint. 

Clearly, the choice of coincidence points is critical for performance, especially when the number 
of points used is small. Alhough some guidelines exist on choosing these points, there is no systematic 
method for the selection. Because the response time of different outputs can vary significantly, coincidence 
points are usually defined separately for each output. 


28.4.3 The Funnel Approach 


The RMPCT algorithm differs from other MPC algorithms in that it attempts to keep each controlled 
output within a user-specified zone called funnel, rather than to keep it on a specific reference trajectory. 
The typical shape of a funnel is displayed in Figure 28.5. The user sets the maximum and minimum limits 
and also the slope of the funnel through a parameter called “performance ratio,” which is the desired time 
to return to the limit zone divided by the open-loop response time. The gap between the maximum and 
minimum can be closed for exact setpoint control, or left open for range control. 

The algorithm solves the following QP at each time: 


P m—1 


min) IIvk + ilk) —y"(kK+ NIG + D> Auk +b (28.65) 
"i=l j=0 
or 
Pp m—1 
min) IIv(k + ilk) —y"(k+ iS + YS luk + jk) — WI (28.66) 
i=l j=0 


subject to usual constraints plus the funnel constraint 
Tink + ilk) <y"(k+ IK) <ynalk ilk), 1<i<p, (28.67) 


where I ink + il|k) and I sank + ilk) represent the upper and lower limit values of the funnel for k + i in 
the prediction horizon as specified at time k. u" is the desired settling value for the input. Note that the 
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FIGURE 28.5 Output penalties used in various formulations. 


reference trajectory y’ is a free parameter, which is optimized to lie within the funnel. Typically, Q>> R 
in order to keep the outputs within the funnel as much as possible. Then one can think of the above as a 
multiobjective optimization, in which the primary objective is to minimize the funnel constraint violation 
by the output and the secondary objective is to minimize the size of input movement (or input deviation 
from the desired settling value in the case of Equation 28.66). In this case, as long as there exists an 
input trajectory that keeps the output within the funnel, the first penalty term will be made exactly zero. 
Typically, there will be an infinite number of solutions that achieve this, leading to a “degenerate” QP. 
The algorithm thus finds the minimum norm solution, which corresponds to the least amount of input 
adjustment—hence the name “Robust” MPCT . However, if there is no input that can keep the output 
within the funnel, the first term will be the primary factor that determines the input. 

The use of funnel is motivated by the fact that, in multivariable systems, the shape of desirable tra- 
jectories for outputs is not always clear due to system interaction. Thus, it is argued that an attractive 
formulation is to let the user specify an acceptable dynamic zone for each output as a funnel and then 
find the minimum size input moves (or inputs with minimum deviation from their desired values) that 
keep the outputs within the zone—or, if not possible, minimize the extent of violation. 


28.4.4 Use of Other Norms 


In defining the objective function, use of norms other than 2-norm is certainly possible. For example, the 
possibility of using 1-norm (sum of absolute values) has been explored to a great extent. Use of infinity 
norm has also been investigated with the aim of minimizing worst-case deviation over time. In both cases, 
one gets an LP, for which a plethora of theories and software exist due to its significance in economics. 
However, one difficulty with these formulations is in tuning. This is because the solution of an LP lies 
at the intersection of binding constraints and it can switch abruptly from one vertex to another as one 
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varies tuning parameters (such as the input weight parameters). The solution behavior of a QP is much 
smoother and therefore, it is a preferred formulation for control. 


28.4.5 Input Parameterization 


In some algorithms such as PFC, the input trajectory can be parameterized using continuous basis 
functions like polynomials. This can be useful if the objective is to follow smooth setpoint trajectories 
precisely, such as in mechanical servo applications, and the sampling time cannot be made sufficiently 
small to allow this with piecewise constant inputs. 

In other commercial algorithms such as HIECON and IDCOM-M, only a single control move is calcu- 
lated,which would correspond to m = 1 in DMC. With this setting, the calculation is greatly simplified. 
On the other hand, use of m= 1 in DMC would limit the closed-loop performance in general. These 
algorithms get around this problem by using a single coincidence point, at which the output is asked to 
match the reference value exactly. 


28.4.6 Model Conditioning 


In multivariable plants, two or more outputs can behave very similarly in response to all the inputs. This 
phenomenon is referred to as “ill-conditioning” and is reflected by a gain matrix that is nearly singular. An 
implication is that it can be very difficult to control these outputs independently with the inputs, as it will 
require an excessive amount of input movement in order to move the outputs in certain directions. Using 
an ill-conditioned process model for control calculation is not recommended as it can lead to numerical 
problems (e.g., inversion of a nearly singular matrix) and also excessive input movements and/or even an 
instability. 

Even though one would check the conditioning of the model at the design stage, because control 
structure can change due to constraints and failures of sensors and actuators, one must make sure at each 
execution time that an ill-conditioned process model is not directly inverted in the input calculation. 

In DMC, direct inversion of an ill-conditioned process model can be circumvented by including a 
substantive input move penalty term, which effectively increases the magnitudes of the diagonal elements 
of the dynamic matrix that is inverted during the least-squares calculation. 

In other algorithms that do not include an input move penalty in the objective function, ill-conditioning 
must be checked at each execution time. In RMPCT, this is done through a method called Singular Value 
Threshholding, where a procedure called “singular value decomposition” is performed on the gain matrix 
to determine those CV directions for which the gain is too low for any effective control at all. Those 
directions with singular values lower than a threshhold value are given up for control and only the 
remaining “high-gain” directions are controlled. SMC-IDCOM addresses this based on the user-defined 
ranking of CVs. Here, whenever an ill-conditioning is detected, CVs are dropped from the control 
calculation in the order of their ranks, starting from the one with the least assigned priority, until the 
condition number improves to an acceptable level. When two CVs are seen to behave very similarly, the 
user can rank the less important CV with a very low priority. Even though the control on the dropped 
CV is given up, it is hoped that it would be controlled indirectly since it behaves similarly to the other 
high-ranked CV. 


28.4.7 Prioritization of CVs and MVs 


In most practical control problems, it is not possible to satisfy all constraints and also drive all outputs 
and inputs to their desired resting values. Hence, priorities need to be assigned to express their relative 
importance. In DMC, these priorities are determined through weight parameters, which enter into the 
various quadratic penalty terms in the objective function. For large, complex problems, determining 
proper weights that lead to an intended behavior can be a daunting task. Even if a set of weights consistent 
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with the control specification is found, the weights can differ vastly in magnitude from one to another, 
causing a numerical conditioning problem. 

Algorithms such as HIECON and SMC-IDCOM attempt to address this difficulty by letting the user 
rank various objectives in the order of their importance. For example, constraint satisfaction may be the 
most critical aspect, which must be taken care of before satisfying other objectives. Also driving the CVs 
to their desired setpoints may be more important than driving the MVs to their most economic values. 
In these algorithms, an optimization would be solved with the most important objective first and then 
remaining degrees of freedom would be used to address the other objectives in the order of priority. These 
algorithms also allow the user to rank each CV and MV according to its priority. Hence, for constraint 
softening, one may specify the order in which constraints for various CVs must be relaxed. Also, in 
setpoint tracking, one can prioritize the CVs so that CVs with higher ranks are driven to their setpoints 
before those with lower ranks are considered. 


28.4.8 Bi-Level Optimization 


The MPC calculation can be split into two parts for an added flexibility. First, a local steady-state 
optimization can be performed to obtain target values for each input and output. This can be followed by 
a dynamic optimization to determine the most desirable dynamic trajectory to these target values. Even 
though the local steady-state optimization can be based on an economic index, it does not replace the 
more comprehensive nonlinear optimization that often runs above the MPC layer—at a much slower 
rate—in order to provide an optimal range of inputs and outputs for the plant condition experienced 
during a particular optimization cycle. The local optimization performed in MPC is based on a linear 
steady-state model, which may be obtained by linearizing a nonlinear model or simply the steady-state 
version of the step response model used in the dynamic optimization. 

The reasons for running the local optimization may vary. For example, one may want to perform 
an economic optimization at a higher frequency to account for local disturbances. Even if there is no 
economic objective in the given control problem, the steady-state optimization can be helpful to determine 
best feasible target values for CVs and the corresponding MV settling values. 

The two-stage optimization can be formulated as below: 


¢ Step 1: Steady-State Optimization The general form of a steady-state prediction equation is 


y(oo|k) = K; (u(co|k) — u(k — 1)) +b(k), (28.68) 
— ee 
Aus(k) 


where y(oo|k) and u(oo|k) are the steady-state values of the output and input projected at time k. 
With only m input moves considered, 


Aus(k) = Au(k) + Au(k+ 1) +--+ Au(k+m-— 1). (28.69) 
Note that, for the step response model, 
yloo|k) = y(k+m-+n-— 1k) (28.70) 


and K; = Sy. Also, 
b(k) = Fin—1(k) + S4Ad(k) + (Yn(k) — Fo(k)). (28.71) 


This steady-state prediction model can be used to optimize a given economic objective function 
subject to various input and output constraints: 


min &(u(oo|k), y(oo|k)). (28.72) 
Aus(k) 
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Since an economic objective function is typically linear and the prediction equation is also linear, 
an LP results. Alternatively, one can also solve 


min ||Au,(k)|I, (28.73) 
Aus(k) 


= k))llo. 28.74 
fae ile yloolk))lla ( ) 


In the first case, we would be looking for a minimum input change such that all the constraints 
are satisfied. In the second case, we would be seek a minimum deviation from the setpoint values 
that are achievable within the given constraints. The solution sets the target settling values for the 
inputs and outputs. 

¢ Step 2: Dynamic Optimization The dynamic prediction equation is the same as before. A quadratic 
regulation objective of the following is minimized subject to the given constraints through QP: 


m+n—2 m—1 
S~ (lk + ilk) — y*(Colk))” QQ(k-+ ilk) — y*(oolk)) + J> Aul (k+jIRAU(K-+jlk) |, 
i=l j=0 
(28.75) 
where y*(oo|k) is the solution from the steady-state optimization. An additional constraint may be 
added to match the settling values of the optimized input trajectories to those computed from the 
steady-state optimization: 


Au(k|k) + Au(k+ 1k) +--+» + Au(k +m — 1k) = Aut (k). (28.76) 


This also forces y(k ++ m+n — 1k) to be at the optimal steady-state value y*(k + oo|k). 


Note that, this steady-state optimization may be performed as often as at every sample time, that is 
at the same execution rate as the dynamic optimization. However, it is critical to filter the noise and 
other high-frequency variations from the feedback signal. Otherwise, the solution from the steady-state 
solution can fluctuate wildly from sample time to sample time, especially in the case of an LP. 


28.5 Future Needs 


28.5.1 Better Identification 


Conventionally, step response models used in DMC (or other industrial MPC algorithms) are identified 
through a series of step tests. In some cases, pseudorandom binary sequence (PRBS) tests instead of step 
tests are used and the step response coefficients are fitted through least squares. In almost all cases, input 
channels are perturbed one at a time, leading to SISO identification. While this practice is simple and 
easy to implement, it does not always yield a multivariable model of required accuracy. This is usually 
dealt with in practice by detuning or dropping a part of control space, not a desirable remedy from a 
performance standpoint. 

Independent testing of input channels emphasizes the accuracy of individual transfer function ele- 
ments, and can result in a poor fit of control-relevant multivariable system characteristics (e.g., the gain 
directionality) [5]. For highly interactive processes, test signals for different input channels need to be 
correlated in order to yield an accurate model for multivariable control. In general, this points to the 
need for a systematic method to design identification experiments with control requirements directly 
considered. In addition, a nonconservative way to quantify the model quality (e.g., uncertainty bounds) 
would greatly aid the much needed integration between identification and control. 

In terms of identification algorithms, development and use of multi-input multi-output (MIMO) 
algorithms that are capable of capturing output correlations can be helpful. They will not only improve 
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prediction in conventional control problems, but will also enable construction of a model useful for 
inferential control. Traditional polynomial-model-based identification algorithms are poorly suited to 
MIMO identification. Recently proposed subspace identification algorithms [8] may fill this need, but 
they need to be tested further and perhaps tailored to suit the process control problems. 


28.5.2 Robust MPC 


Model uncertainty is an important aspect of every process control problem, as most processes are very 
difficult and time consuming to model accurately. Hence, it is desirable to incorporate the model quality 
information (e.g., uncertainty bounds, parameter probability distribution) directly into the control com- 
putation, instead of achieving robustness margins indirectly through various tuning parameters. In terms 
of the robust MPC controller synthesis, the most popular approach has been the so-called min-max 
predictive control that aims at minimizing the worst-case error with respect to the model set. Other 
methods include a stochastic approach where the expectation of the error is minimized, given a certain 
probability distribution of model parameters [7]. None of these methods have been tested in real prob- 
lems. See [2] for a review and references. In order for these methods to find practical applications, what is 
most needed is a method that reliably computes the uncertainty bounds or probability distribution from 
identification data. 


28.5.3 Performance Monitoring, Diagnosis, and Adaptation 


It has been reported that many MPCs perform well when commissioned, but their performances deteri- 
orate over time and they eventually have to be taken offline. There are a wide variety of causes for this, 
including instrumentation problems, process nonlinearity, and parameter variations. In order to sustain 
the benefits of these controllers over a long period of time, a mechanism to detect and diagnose the cause 
of significant performance deterioration is needed. The results can be communicated to engineers and 
can also be used to adapt control parameters. Vigorous research has already begun in this area and some 
vendors (such as Honeywell) have expressed interests in including such features in the next generation of 
commercial software. 


28.6 Conclusion 


This chapter presented a brief review and outlook of MPC in the process industries. The original industrial 
algorithms, albeit simple and restrictive, are continuing to be used quite successfully in process control 
applications. Some generalizations and extensions introduced mostly by academics to enhance the original 
industrial technology have found their way into the next generation of commercial algorithms being 
marketed currently. This indicates a healthy synergism between academia and industry in this particular 
area. This chapter also provided some needs (in both research and practice) to make MPC a more 
appealing and broadly applicable technique in the future. 
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29.1 Introduction 


Traditional control theory is concerned with the design of linear feedback control with desirable 
asymptotic behavior, such as stability and small steady-state and tracking errors, while properties of 
transient behavior are expressed in terms of overshoot and speed of response. External disturbances can 
be handled by modeling these as random processes, leading to the Linear Quadratic Gaussian (LQG) 
problem formulation. This theory has some limitations. 

Because the feedback law is specified to be linear, it is not possible for design methods to explicitly 
incorporate hard bounds on the control values, for example, the requirement that the applied force should 
not exceed a specified limit. Second, it is not possible to express finite time requirements, for example, 
the requirement that the system state reach a prespecified value at a prespecified time. Third, it is not 
possible to demand guaranteed performance in the face of disturbances, for example, the requirement 
that a certain target state be reached, no matter what the disturbance. Thus control problems with hard 
bounds on the control values, restrictions on the state trajectory over a finite time horizon, and guaranteed 
behavior despite the disturbances, are difficult to solve using frequency-based design methods. 

In order to address these problems one needs to study system evolution in the time domain. The central 
concept that emerges in such studies is that of the reach set, which is the set of states that can be reached 
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by using all possible controls. This chapter is devoted to the formulation and computation of the reach 
set of a linear system with disturbances. The concept of reachability was introduced in [28]; [29] shows 
the reach set can be computed by solving the forward Hamilton-Jacobi-Bellman-Isaacs (HJBI) partial 
differential equation; and the notion of backward reachability with its application to aiming at a specified 
target set is described in [20]. Reachability of hybrid systems is addressed in [30,35]. Over the last decade, 
significant advances were made in the characterization of reach sets and their computation for linear 
systems. These advances are described in this chapter. 

Section 29.2 introduces the forward and backward reach sets, the classes of open- and closed-loop 
controls, and different kinds of reach sets that are appropriate in dealing with disturbances. Although 
some of the discussion applies to nonlinear systems, explicit formulas for reach sets are available only for 
linear systems. These formulas lead to explicit algorithms for computing reach sets (of linear systems) 
and Section 29.3 critically reviews the most promising algorithms. Section 29.4 is devoted to a set of 
algorithms based on the ellipsoidal calculus. These algorithms have a lower computational complexity, 
greater accuracy, and can work with systems of a larger size, compared with those reviewed in Section 
29.3. Finally, Section 29.5 presents three examples to illustrate the ellipsoidal-based approach. 


29.2 Basics of Reachability Analysis 


29.2.1 Systems without Disturbances 


Consider a general continuous-time 
x(t) =f (t,x, 4), (29.1) 


or discrete-time dynamical system 
x(t+1)=f(t,x,u), (29.1d) 


wherein ¢ is time*, x € R” is the state, u € R™ is the control, and f is a measurable vector function taking 
values in R”.* The control values u(t, x(t)) are restricted to a closed compact control set U/(t) C R™. An 
open-loop control does not depend on the state, u = u(t); for a closed-loop control, u = u(t, x(t)). 


Definition 29.1: Reach Set 


The (forward) reach set V(t, to, xo) at time t > to from the initial position (to, xo) is the set of all states 
x(t) reachable at time t by system (Equation 29.1, or 29.1d), with x(to) = xo through all possible controls 
u(t, x(t)) € U(t), to < t < t. Fora given set of initial states Xo, the reach set V(t, to, Xo) is 


X(t; to) = J X(t to,x0). 


xpEXo 


Here are two facts about forward reach sets. 


1. X(t, to, Xo) is the same for open-loop and closed-loop control. 
2. X(t, to, Xo) satisfies the semigroup property, 


X(t, to, Xo) = X(t, X(t, to, X)), to<t<t. (29.2) 


In discrete-time case, t assumes integer values. 

+ We are being general when giving the basic definitions. However, it is important to understand that for any specific 
continuous-time dynamical system it must be determined whether the solution exists and is unique, and in which class of 
solutions these conditions are met. we shall assume that function f is such that the solution of the differential equation 
29.1 exists and is unique in Fillipov sense. This allows the right-hand side to be discontinuous. For discrete-time systems, 
this problem does not exist. 
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For linear systems 
f(t.x, u) = A(t)x(t) + B(du, (29.3) 


with matrices A(t) in R?*” and B(t) in R”*". For a continuous-time linear system, the state transition 
matrix is 
D(t, to) = A(t) P(t, to), PO(t,t) =I, 


which for constant A(t) = A simplifies as 
®(t, ty) = eA), 
For a discrete-time linear system, the state transition matrix is 
P(t +1,t) =AHP(Et), O(6t=1, 
which for constant A(t) = A simplifies as 
P(t, ty) =A”. 


If the state transition matrix is invertible, ®~!(t, to) = ®(to, t). The transition matrix is always invert- 

ible for continuous-time and for sampled discrete-time systems. However, if for some t, to < t < ft, A(t) 

is degenerate (singular), ®(t, to) = ies A(t) is also degenerate and cannot be inverted. 
Following Cauchy’s formula, the reach set V(t, to, Xo) for a linear system can be expressed as 


t 
X(t, to, Xo) = P(t, to) Xo ef P(t, t)B(t)U(t) dt (29.4) 


to 
in continuous-time, and as 
t-1 
X(t, to, Xo) = P(t, to) Xo B Y> O(t, 1+ 1I)B(DU(t) (29.4d) 
t=to 
in discrete-time case. 
The operation “@” is the geometric sum, also known as Minkowski sum.* The geometric sum and 


linear (or affine) transformations preserve compactness and convexity. Hence, if the initial set % and the 
control sets U/(t), fo < t < t, are compact and convex, so is the reach set V(t, to, Xo). 


Definition 29.2: Backward Reach Set 


The backward reach set V(t, t, y1) for the target position (t,, y:) is the set of all states y(t) for which there 
exists some control u(t, x(t)) € U(t), t < t < t, that steers system (Equation 29.1 or 29.1d) to the state y, 
at time t). For the target set V at time ty, the backward reach set V(t), t, V1) is 


V(t, t,V1) = U V(t, t.y1). 


nevi 


The backward reach set Y(t, t, 1) is the largest weakly invariant set with respect to the target set V1 
and time values t and ft." 


Minkowski sum of sets W, Z C R” is defined as W@ Z = {w+z|weW, ze Z}. Set W @ Z is nonempty if and only 
if both W and Z are nonempty. If W and Z are convex, set W @ Z is convex. 

M is weakly invariant with respect to the target set Y) and times fo and t, if for every state x9 € M there exists a control 
u(t, x(t)) €U(t), to < t <t, that steers the system from xq at time fp to some state in at time tf. If all controls in 
U(t), to < t < t steer the system from every xo € M at time fo to )) at time t, set M is said to be strongly invariant with 
respect to }, to, and f. 
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Remark 29.1 


Backward reach set can be computed for continuous-time system only if the solution of Equation 29.1 
exists for t < t,; and for discrete-time system only if the right-hand side of Equation 29.1d is invertible*. 
These two facts about the backward reach set Y are similar to those for forward reach sets. 


1. Y(t), t, 1) is the same for open-loop and closed-loop control. 
2. V(t1,t, V1) satisfies the semigroup property, 


V(t, t,V1) = V(t, t, Vt 1), t<t<h. (29.5) 


For the linear system (Equation 29.3) the backward reach set can be expressed as 


t 


Vet t.d1) = Olt h) ® i (t,)B(OU(t) de (29.6) 


ty 
in the continuous-time case, and as 
ty-1 


V(t, t, Vi) = Ot, V1 @ D> — P(t, t)B(U(a) (29.6d) 


tt 
in the discrete-time case. The last formula makes sense only for discrete-time linear systems with invertible 
state transition matrix. Degenerate discrete-time linear systems have unbounded backward reach sets and 
such sets cannot be computed with available software tools. 

Just as in the case of forward reach set, the backward reach set of a linear system Y(t), t, V1) is compact 
and convex if the target set J, and the control sets U/(t), t < t < t; are compact and convex. 


Remark 29.2 


In the computer science literature, the reach set is said to be the result of operator post, and the backward 
reach set is the result of operator pre. In the control literature, the backward reach set is also called the 
solvability set. 


29.2.2 Systems with Disturbances 
Consider the continuous-time dynamical system with disturbance 
x(t) =f (t,x, u,v), (29.7) 
or the discrete-time dynamical system with disturbance 
x(t+ 1) =f (t,x, u,v), (29.7d) 


in which we also have the disturbance input v € R@ with values v(t) restricted to a closed compact set 
V(t) C R¢. 

In the presence of disturbances, the open-loop reach set (OLRS) is different from the closed-loop reach 
set (CLRS). 

Given the initial time fo, the set of initial states Vp, and terminal time f, there are two types of OLRS. 


Definition 29.3: OLRS of Maxmin Type 


The maxmin open-loop reach set Xox(t, to, Xo) is the set of all states x, such that for any disturbance 
v(t) € V(t), there exists an initial state x9 € Xp and a control u(t) € U(t), to < t < t, that steers system 
(Equation 29.7 or 29.7d) from x(to) = xo to x(t) = x. 


* There exists f—1(t, x, u) such that x(t) = f—l(t, x(t +1), u,v). 
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Definition 29.4: OLRS of Minmax Type 


The minmax open-loop reach set Xo, (t, to, Xo) is the set of all states x, such that there exists a control 
u(t) € U(t) that for all disturbances v(t) € V(t), to < t < t, assigns an initial state x9 € Xo and steers 
system (Equation 29.7 or 29.7d), from x(t) = xo to x(t) = x. 


In the maxmin case, the control is chosen after knowing the disturbance over the entire time interval 
[to, t], whereas in the minmax case, the control is chosen before any knowledge of the disturbance. 
Consequently, the OLRS do not satisfy the semigroup property. 

The terms “maxmin” and “minmax” come from the fact that Voz(t, to, Xo) is the subzero level set of 
the value function 


V(t,x) = max min{dist(x(to), Xo) | x(t) =x, u(t) €U(t), v(t) € V(t), to < t < t}, (29.8) 


that is., Voz(t, to, Xo) = {x | V(t, x) < 0}, and Xoz(t to, Vo) is the subzero level set of the value function 


Vi(t,x) = min max{dist(x(to), Xo) | x(t) =x, u(t) €U(t), v(t) € V(t), to < t < t}, (29.9) 


in which dist(-,-) denotes Hausdorff semidistance.* Since V(t,x) < V(t,x), Xo, (t, to, Xo) C Vorlt, 
to, Xo). 

Note that maxmin and minmax OLRS imply guarantees: these are states that can be reached no matter 
what the disturbance is, whether it is known in advance (maxmin case) or not (minmax case). The OLRS 
may be empty. 

Fixing time instant 11, to < 11 < t, define the piecewise maxmin open-loop reach set with one correction, 


More to, Xo) = Xox(t, 1, Vor(t to, Xo); (29.10) 
and the piecewise minmax open-loop reach set with one correction, 
X(t, to Xo) = Xo (bu, Xz (1 to, X%)). (29.11) 
The piecewise maxmin OLRS Re i (t, to, Xo) is the subzero level set of the value function 
vi(t,x) = max min{V(t1,x(t1)) | x(t) =x, u(t) €U(t), v(t) € V(t), T1 < Tt < th, (29.12) 
with V(t1,x(t1)) given by Equation 29.8, which yields 
VIG) = VG x), 


and thus, ; 
Koz (ts tov) © Kor(t, to, Xo). 


On the other hand, the piecewise minmax OLRS %, Le, to, Xo) is the subzero level set of the value 
function 


Vi (t,x) = min max{V(t1, x(t1)) | x(t) =x, u(t) €U(t), v(t) € V(t), T1 < T < th, (29.13) 


* Hausdorff semidistance between compact sets W, Z C R" is defined as 


dist(W, Z) = min{(w — z,w — z)i/2 |wew, ze Z}, 


where (-,-) denotes the inner product. 
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with V(t), x(t1)) given by Equation 29.9, which yields 
V(t,x) = V(t), 


and thus, 
Xoz(t, too) © Xz (ts to, Xo). 


We can now recursively define piecewise maxmin and minmax OLRS with k corrections for to) < t1 <--- 


<1] < t. The maxmin piecewise OLRS with k corrections is 
=k = Sk-1 
Xor (ts to Yo) = Kort tes Kop (th tos Xo), 


which is the subzero level set of the corresponding value function 


v* (t,x) = max min{ V*~" (tq, x(t) | x(t) =x, u(t) €U(t), v(t) E V(t), TH << Tt}. 


The minmax piecewise OLRS with k corrections is 
1 (ts fos Xo) = Lor (ts te XGr' (the fo» %0)) 


which is the subzero level set of the corresponding value function 


Vet, x) = min max{V* (14, x(x;)) | x(t) =x, u(t) €U(t), v(t) € V(t), TH <T< th. 


From Equations 29.12, 29.13, 29.15, and 29.17 it follows that, 


Vi(t,x) < V(t,x) <--» < V*(t,x) < Viit,x) <-- < Vix) < Vit»). 


Hence, 
Kort 4) G2 ty) SOC AE ty, A) 
ak | a 
c Xozlts to, Xo) Cee Xo (ts to, X%) c Kort, to, X). 
We call 


oo for continuous-time system, 


= sk 
Xcx(t, to, Xo) = Xo, (t,to, Xo), k= ; : 
cult to, +o) ott fo, to) ( —to—1 for discrete-time system, 


the maxmin closed-loop reach set of system (Equation 29.7 or 29.7d) at time t, and we call 


oo for continuous-time system, 


k 
X oy (t to, Xo) = XO (ts to. Xo), k= : : 
Xx (t; to, ¥o) = Kor (ts to, +o) ( —to—1 for discrete-time system, 


the minmax closed-loop reach set of system (Equation 29.7 or 29.7d) at time f. 


Definition 29.5: CLRS of Maxmin Type 


(29.14) 


(29.15) 


(29.16) 


(29.17) 


(29.18) 


(29.19) 


(29.20) 


Given initial time to and the set of initial states Xo, the maxmin CLRS X cx (t, to, Xo) of system (Equation 29.7 
or 29.7d) at time t > to, is the set of all states x, for each of which and for every disturbance v(t) € V(t), there 
exist an initial state x9 € Xo and a control u(t, x(t)) €U(t), such that the trajectory x(t|v(t), u(t, x(t))) 


satisfying x(t) = xo and 
X(t|v(t), u(t, x(t))) € f(t, x(t), u(t, x(t), v(t) 
in the continuous-time case, or 
x(t + 1] v(t), u(t, x(t))) € f(t, x(t), u(t, x(t), v(t) 


in the discrete-time case, with to < t < t, is such that x(t) = x. 
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Definition 29.6: CLRS of Minmax Type 


Given initial time to and the set of initial states Xo, the maxmin CLRS X cx (t, to, Xo) of system (Equation 
29.7 or 29.7d), at time t > to is the set of all states x, for each of which there exists a control u(t, x(t)) € 
U(t), and for every disturbance v(t) € V(t) there exists an initial state x9 € Vp, such that the trajectory 
x(t, v(t)|u(t, x(t))) satisfying x(to) = xo and 


X(t, v(t) u(t, x(t))) € f(t, x(t), u(t, x(t), v(t) 
in the continuous-time case, or 
x(t + 1, v(t)|u(t, x(t))) € f(t, x(t), u(t, x(t), v(T)) 


in the discrete-time case, with ty < t < t, is such that x(t) = x. 


By construction, both maxmin and minmax CLRS satisfy the semigroup property (Equation 29.2). 
For some classes of dynamical systems and some types of constraints on initial conditions, controls and 
disturbances, the maxmin and minmax CLRS may coincide. This is the case for continuous-time linear 
systems with convex compact bounds on the initial set, controls and disturbances under the condition 
that the initial set Yo is large enough to ensure that V(to + €, to, Yo) is nonempty for some small € > 0. 
Consider the linear system case, 


f (t,x, u) = A(t)x(t) + B(t)u+ G(t)v, (29.21) 


where A(t) and B(f) are as in Equation 29.3, and G(t) takes its values in R¢, 
The maxmin OLRS for the continuous-time linear system can be expressed through set valued integrals, 


t t 
Xox(t, to, Xo) = (ou. to) Xo ® / P(t, t)B(t)U(t) ar) = i ®(t, t)(—G(t)) V(t) dt, (29.22) 
to to 


and for discrete-time linear system through set-valued sums, 


t-1 t-l1 
Xor(ts to, Xo) = (2 to) Xo ® oS @(t,t+ paca) = > @(t,t+1)(—G(t))V(t). (29.22) 


t=1to t=1t0 


Similarly, the minmax OLRS for the continuous-time linear system is 
t t 
Xoz(t to, Xo) = (ow. ty) Xo— / P(t, t)(—G(t)) V(t) ax) @ i P(t, t)B(t)U(t) dt, (29.23) 
to to 


and for the discrete-time linear system, it is 


t-1 t—1 
X op (t, to, Xo) = (v6 to)Xo~ D> O(t, t+ ni-ato)vt0) ® Y\ (t+ IBOU(t). (29.234) 


t=lo t=lo 


The operation ‘—’ is geometric difference, also known as Minkowski difference.* 


* The Minkowski difference of sets W,Z € R” is defined as W—Z = {g ER"|E@ZC w}. If W and Z are convex, 
W-2Z is convex if it is nonempty. 
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Now consider the piecewise OLRS with k corrections. Expression 29.14 translates into 


t 


t 
Rout to, Xo) = (et. 1)%6r'(einto, 0) f P(E, t)B(t)U(t) ar) -| P(E, t)(—G(t)) V(t) dt, 


k k 
(29.24) 


in the continuous-time case, and for the discrete-time case into 


t-1 t-1 
Roe to, Xo) = (2 14) X61, (thy to, Xo) ® oe P(t, t+ paca) = S P(t, t+ 1)(—G(t))V(t). 


TH=Tk THTk 


(29.24d) 
Expression 29.16 translates into 


t t 
X61 (ts to, Xo) = (ee naiites too) ~ f oe n-Ger@ar) of P(t, t)B(t)U(t) dt, 


Tk k 
(29.25) 
in the continuous-time case, and for the discrete-time case into 


t-1 t—l1 
X61 (ts tos Xo) = (ve TAG (th to, Xo) — J) O(t. + ni-ato)vt0) ® D> Ot, t+ 1B(DU(A). 


THT THTk 


(29.25d) 
Since for any W1, W2, W3 C R” it is true that 


(Wi —W2) B W3 = (Wi ® W3)—(W2 B W3) © (Wi @ W3)—W), 


from Equations 29.24 and 29.25 and from Equations 29.24d and 29.25d, it is clear that Equation 29.18 is 
true. 

For linear systems, if the initial set >, control bounds 7/(t) and disturbance bounds V(t), to < t < t, 
are compact and convex, the CLRS Vcr(t, to, Xo) and ¥ cy(t to, Xo) are compact and convex, provided 
they are nonempty. For continuous-time linear systems, ¥cr(t, fo, Xo) = Xcr(t, to, Xo) = Xcx(t, to, X0). 

Just as for forward reachability case, the backward reach sets can be open-loop (OLBRS) or closed-loop 
(CLBRS). 


Definition 29.7: OLBRS of Maxmin Type 


Given the terminal time t, and target set Y, the maxmin open-loop backward reach set Vor(t ts 1) of 
system (Equation 29.7 or 29.7d) at time t < t, is the set of all y, such that for any disturbance v(t) € V(t) 
there exists a terminal state y, € VY, and control u(t) €U(t), t << t, which steers the system from 


y(t) = y to y(t) = y1. 
Vor(t, t, V1) is the subzero level set of the value function 


Vi, (ty) = max min{dist(y(t1), V1) | y(t)=y, u(t) €U(t), v(t) € V(t), t< T< ty}, (29.26) 


Definition 29.8: OLBRS of Minmax Type 


Given the terminal time t, and target set , the minmax open-loop backward reach set Yor (tt t, V1) of 
system (Equation 29.7 or 29.7d) at time t < t, is the set of all y, such that there exists a control u(t) € U(t) 
that for all disturbances v(t € V(t), t < t < ft, assigns a terminal state y; € Y, and steers the system from 
y(t) = y to y(t) = 1. 
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y PAGE t, Y,) is the subzero level set of the value function 


Vi(ty) = min max{dist(y(t1), V1) | y(t)=y, u(t) €U(t), v(t) € V(t), t< T< ty}, (29.27) 


Remark 29.3 


The backward reach set can be computed for a continuous-time system only if the solution of Equation 
29.7 exists for t < t,, and for a discrete-time system, only if the right-hand side of Equation 29.7d is 
invertible. 

Similar to the forward reachability case, we construct piecewise OLBRS with one correction at time 11, 
t <1 < t,. The piecewise maxmin OLBRS with one correction is 


Vor (ths t,.Vi) = Vor(t15 ts Vou(ts 4 V1); (29.28) 
and it is the subzero level set of the function 
Vi (by) = max min{V,(t1,y(t1)) | y(t) = ys u(t) € U(t), v(t) € V(t), EST < Ti}. (29.29) 
The piecewise minmax OLBRS with one correction is 
Voz (tre ts Vi) = Voy (ta ts Voy (ty 1.1), (29.30) 
and it is the subzero level set of the function 


Vi (ty) = min max{Vp(t1, y(1)) | y(t) =y, u(t) €U(t), v(t) € V(t), f< T< Ty}, (29.31) 


Recursively define maxmin and minmax OLBRS with k corrections for t< 1% <---<t™, <t. The 
maxmin OLBRS with k corrections is 


Vout. .01) = Vor (test Vor (tr te). (29.32) 
which is the subzero level set of function 
Vi(ty) = max min{V5 | (ths y(t) | y(t) =y, u(t) €U(t), v(t) E V(t), << Th}. (29.33) 
The minmax OLBRS with k corrections is 
Vir (ob) = Vo, (te 6 VE Mt wD), (29.34) 
which is the subzero level set of the function 
Vilt,y) — min max{V5 (te y(t4)) | y(t) =y, u(t) €U(t), v(t) E V(t), t< Tt < Th}, (29.35) 
From Equations 29.29, 29.31, 29.33, and 29.35 it follows that 


=H a per 


Hence, 
Voz (tri) SV), (ti) G+ CVE (4.6.1) 
© Vor (t1,t,01) S +S Vor lt 1) S Voult tN). (29.36) 
We say that 


oo for continuous-time system 


: ; 29.37 
t; -t—1 for discrete-time system ( 


Veatch Y= Verte tovs k=| 


is the maxmin closed-loop backward reach set of system (Equation 29.7 or 29.7d) at time t. We say that 


oo for continuous-time system 


; : 29.38 
t; -t—1 for discrete-time system ( ) 


Vor (tt Vi) = V5, (tb Vi), k= | 


is the minmax closed-loop backward reach set of system (Equation 29.7 or 29.7d) at time f. 


29-10 Control System Advanced Methods 


Definition 29.9: CLBRS of Maxmin Type 


Given the terminal time ty and target set 1, the maxmin CLBRS Ycz (ti, t, Vi) of system (Equation 29.7 or 
29.7d) at time t < ty is the set of all states y, for each of which for every disturbance v(t) € V(t) there exists 
terminal state y, € Y, and control u(t, y(t)) € U(t) that assigns trajectory y(t, |v(t), u(t, y(t))) satisfying 


(t|v(z), u(t, y(t))) € f(t, y(t), u(t, y(t), V(t) 
in continuous-time case, or 
Wr+ I v(t), u(t, y(t) € f(t, y(t), u(t, y(t), V(T)) 


in discrete-time case, with t < t < t), such that y(t) = y and y(t)) = y1. 


Definition 29.10: CLBRS of Minmax Type 


Given the terminal time t, and target set V,, the minmax CLBRS Vor (ts t, V1) of system (Equation 29.7 or 
29.7d) at time t < ty is the set of all states y, for each of which there exists control u(t, y(t)) € U(t) that for 
every disturbance v(t) € V(t) assigns terminal state y, € )\ and trajectory y(t, v(t)|u(t, y(t))) satisfying 


Ie v(v)| u(t, y(T))) € F(t y(t), Ut y(T)), V(T)) 
in the continuous-time case, or 
y(t + 1, v(t) u(t, y(t))) € f(t y(), u(t, y(1)); V(T)) 
in the discrete-time case, with t < t < ty, such that y(t) = y and y(t) = y1. 
Both maxmin and minmax CLBRS satisfy the semigroup property (Equation 29.5). 


The maxmin OLBRS for the continuous-time linear system can be expressed through set valued 
integrals, 


t ty 
Tite ons (eumne / (6 BME) dt) ~ / Ot, NGOV(t)dt, (29.39) 
ty t 


and for the discrete-time linear system through set-valued sums, 


ty-1 tj-1 
Yorlt.t01) = (2 t)1 ® Y) — P(t, t+ paca) ~S0 Ot, t+)G()V(r). (29.394) 


tt tt 


Similarly, the minmax OLBRS for the continuous-time linear system is 


ty t 
Volt Vi) = (ou. t)— / ®(t, 1)G(t)V(t) a)e i ®(t, 1)B(U(t) dt, (29.40) 
t hy 


and for the discrete-time linear system it is 


tj-1 ty-1 
Voz (tot Wi) = (+ t)Vi— S O(t,t+ Hacer ® > —@(t,t+1)B(t)U(t). (29.40) 


tt tt 
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Now consider piecewise OLBRS with k corrections. Expression 29.32 translates into 


t T 
V(t.) = (oe. brn we / (t,)B(DU(a) ax) & / * B(t, )G()V() da, 
Tt t 


k 


(29.41) 
in the continuous-time case, and for the discrete-time case into 
‘ ; hH%-1 %—1 
= —k-1 ; 
Voto tN) = | OG WHEL (te wIV® Do —O(, 1+ DBM) | = 7 (44+ NEVO. 
tt t=t 
(29.41d) 


Expression 29.34 translates into 


T t 
YE (tt) = (ow tH) or (ts te Va) / * O(, DG(DV(A) ar) ® i (t, 1)B(x)U(1) dr, 
t T 


k 


(29.42) 
in the continuous-time case, and for the discrete-time case into 
4-1 4-1 
VE tot = (OC, WIG. (4 HII= D> O(,1+ NEVA) | © D> - O14 DBOU(0). 
. “ (25.42d) 


For continuous-time linear systems Yer (ty, t, V1) = Voz ltr, t, V1) = Ver (th, t, V1) under the condition 
that the target set Y is large enough to ensure that Y cy {ti t, — €, 1) is nonempty for some small € > 0. 

Computation of backward reach sets for discrete-time linear systems makes sense only if the state 
transition matrix ®(ty, t) is invertible. 

If the target set V1, control sets /(t), and disturbance sets V(t), t < t < tj, are compact and convex, 
then CLBRS Vcr (ty, t; V1) and y cyt t, Y,) are compact and convex, if they are nonempty. 


29.2.3 Reachability Problem 


Reachability analysis is concerned with the computation of the forward “V(t, to, ¥o) and backward 
V(t, t, Vi) reach sets (the reach sets may be maxmin or minmax) in a way that can effectively meet 
requests like the following: 


1. For the given time interval [to, t], determine whether the system can be steered into the given target 
set Vj. In other words, is the set YN Us, <r<tt (t, to, Yo) nonempty? And if the answer is “yes,” 
find a control that steers the system to the target set (or avoids the target set).* 

2. Ifthe target set Yj is reachable from the given initial condition {to, Yo} in the time interval [to, f], 
find the shortest time to reach J, 


arg min{.V(t, to, Vo) NV AG| to <t< th. 
T 


3. Given the terminal time f,, target set Y), and time t < ty, find the set of states starting at time ¢t from 
which the system can reach ) within time interval [t, t)]. In other words, find L), Stet V(t, 1 V1). 

4. Find a closed-loop control that steers a system with disturbances to the given target set in given 
time. 

5. Graphically display the projection of the reach set along any specified two- or three-dimensional 
subspace. 


So-called verification problems often consist in ensuring that the system is unable to reach an “unsafe” target set within a 
given time interval. 
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For linear systems, if the initial set Vo, target set V1, control bounds 2/(-), and disturbance bounds V(-) 
are compact and convex, so are the forward X(t, to, Yo) and backward Y(t), t, 1) reach sets. Hence 
reachability analysis requires the computationally effective manipulation of convex sets, and performing 
the set-valued operations of unions, intersections, geometric sums, and differences. 

Existing reach set computation tools can deal reliably only with linear systems with convex constraints. 
A claim that a certain tool or method can be used effectively for nonlinear systems must be treated with 
caution, and the first question to ask is for what class of nonlinear systems and with what limit on the 
state space dimension does this tool work? Some “reachability methods for nonlinear systems” reduce to 
the local linearization of a system followed by the use of well-tested techniques for linear system reach set 
computation. Thus these approaches in fact use reachability methods for linear systems. 


29.3 Overview of Computational Methods and Tools 


Before choosing a method and a tool for reachability analysis, one should answer the following questions 
to specify the requirements. 


1. Do you really need to compute reach sets, or it is enough to perform a safety check, for example, 
to ensure that trajectories of a system never enter a given target set, or never leave a given initial 
set? Barrier functions or invariant sets, described in the end of this section, may be sufficient for 
safety checking. 

2. Do you need to compute reach sets exactly, or will approximations, external and internal, be 
enough? Except for very specific classes of systems, exact reach set computation is not possible, 
and approximation techniques are required. Unless a reach set has simple structure, its exact 
representation is possible only for low state-space dimension. Hence, the next question. 

3. What is the dimension of your system? The higher the system dimension, the rougher the reach 
set approximation. 


Another important quality of a computational method for reach sets is the preservation of the semi- 
group property. It is highly desirable that the semigroup property is maintained by the algorithm as well 
as by its software implementation. 


29.3.1 Level Set Method 


We start the overview of computational techniques for reach sets with the level set method as it points 
out the essence of the reachability problem, and has been used in practice for specific nonlinear systems. 
The idea is to solve the HJBI partial differential equation 


OV 
ae + max (2% ft w) =0, 


with initial condition 
V (to, x) = dist(x, Xo), 


for t > to, and then to find the reach set V(t, to, Xo) as the subzero level set of the solution V(t, x), 
X(t, to, Xo) = {x ER" | V(t,x) <0}. 
This forward HJBI equation was introduced in [29]. For systems with disturbances, the HJBI equation is 


ov i ov (t ))=0 
pp + min max ac »X,U, V)) = 
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in the maxmin case, and _ _ 
OV z _ [OV fee er, 
— + max min ( —,f(t,x,u,v)) = 
ot uv \ Ox 
in the minmax case. 
For backward reach sets, the HJBI equation is solved in backward time, 


OV, OV, 
as + min (2°. pee “). 
with boundary condition 

Vo(tr, x) = dist(x, VY ), 


for t < t). For systems with disturbances, the backward HJBI is 


in the maxmin case, and 

OV, < OV, 

aE + min max (Aes ices u, ») =0 
in the minmax case. 

Computation of reach sets as level sets of HJBI solutions was introduced in [21,22,24] with special 
emphasis on linear systems. In [31], the authors applied the level set method to reachability analysis of 
hybrid systems. The level set method is implemented in the Level Set Toolbox [5], which uses numerical 
algorithms for time-dependent HJBI equations and structured grids. Work is under way to implement 
fast marching methods. These are effective numerical schemes that work for time-independent HJBI, but 
whose major restriction is the need for the control to have the same dimension as the state. Level Set 
Toolbox tries to compute the surface of the reach set exactly with accuracy dependent on the choice of the 
grid. This plus the exponential growth of computational complexity with the system dimension makes 
the level set method impractical for systems with dimension larger than three. 

Level Set Toolbox deals with continuous-time systems. To use the level set method in discrete-time case, 
one has to solve the Bellman equation under the condition that the right-hand side of system (Equation 
29.1d) is invertible, 

Vit+tla= min(V(t,f~"(t,% u))) 


with initial condition 
V (to, x) = dist(x, Xo) 


for t > to, and then find the forward reach set 
X(t, to, Xo) = {x € R" | V(t,x) < 0}. 


The backward reach set is the subzero level set of the value function V;,(t, x) obtained from the backward 
Bellman equation, 
V,(t — 1, x) = min(V,(t, f(t — 1, x, u))), 


with boundary condition 
Vo(tr, x) = dist(x, VY ), 


for t < t,. For systems with disturbances min, is substituted with max, min, or with min, max, in both 
forward and backward Bellman equations, and the functions f and f—!, whose existence is required, 
depend on an additional parameter v. 


29-14 Control System Advanced Methods 


Even though in the discrete-time case the computation of the value function does not involve solving 
a partial differential equation (PDE), it is still very burdensome, especially for nonlinear systems whose 
reach sets are nonconvex. Computing the distance function for such sets and minimizing it over u may 
be difficult. Even more difficult it is to search for maxmin or minmax. 

The conclusion is that although the level set method handles nonlinear systems, it is computationally 
costly, and the need to maintain a grid with the value function values, which must be rather dense to 
ensure proper accuracy, makes it practical only for low-dimensional dynamical systems. 


29.3.2 Quantifier Elimination 


For some classes of systems the reach sets can be computed symbolically using quantifier elimination. 
Quantifier elimination is the removal of all quantifiers (the universal quantifier V and the existential quan- 
tifier 4) from a quantified system. Each quantified formula is substituted with quantifier-free expression 
with operations +, x, = and <. For example, consider the discrete-time linear system (Equations 29.1d, 
SO <a 
10 30 
controls u(t) € {ue R| —1 <u <1}, the reach set for t > 0 is given by the quantified formula 


29.3), with A(t) = A < 


and B(t) = B= : . For initial conditions x9 € {x € R? | ||x|loo < 1} and 


t-1 
€ R? | 5x0, at > 0,3u(t),0<t<t:x=Alxot yarmtmue} ‘ 


t=0 
which is equivalent to the quantifier-free expression 
—1<[l 0lx<1A —1<[0 1]x<1. 


It is proved in [27] that if A is constant and nilpotent or is diagonalizable with rational real or purely 
imaginary eigenvalues, the quantifier elimination package returns a quantifier free formula describing 
the reach set. This class of systems is evidently rather limited. 

Requiem [10] is a Mathematica notebook which, given a linear system, the set of initial conditions and 
control bounds, symbolically computes the exact reach set, using the experimental quantifier elimination 
package. 


29.3.3 Polytope Method 


Definition 29.11: Hyperplane 


The hyperplane H(c, y) in R" is the set 
H={xeR"| (x) =y} 
with fixedc € R" andy ER. 


A hyperplane defines two (closed) halfspaces, 
S; = {x ER" | (c,x) <y}, and S)={xER"| (c,x) > y}. 


Definition 29.12: Polytope 


The polytope P(C, g) is the intersection of a finite number of closed halfspaces: 
P= {xe R"|Cx <g}, 
with fixed C =[c, --- cm]? €R™*” andg=[y1 --- Ym) €R"™. 
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For linear discrete-time systems (Equations 29.7d and 29.21), with X,U(t) and V(t), t > to, being 
polytopes in R", R™, and R@ respectively, the reach sets Vcr(t, fo, Xo) and ¥¢z (t, to, Xo) are also polytopes, 
because the polytope structure is closed under the operations of affine transformation, geometric sum, 
and geometric difference. (For continuous-time systems the reach sets need not be polytopes.) Starting 
with initial condition 

X cx (to, tor Xo) = X cz (to. to, Xo) = Xo» 


for time step t > fo these reach sets are computed exactly as 


Xcx(t to, Xo) = A(t — I) X(t — 1, to, Xo) © B(t — DU(t — 1I)—G(t — 1) V(t — 1), 


and 
X cr (ts to, Xo) = A(t — YA (t — 1, to, X)—G(t — IV(t — 1) @ Bt — DU(t — VD. 


A similar procedure works for backward reach sets if the matrices A(t), t < t; are nondegenerate. 

As we see, the polytope method consists in sequential computation of affine transformations of poly- 
topes, geometric sum of two polytopes, and geometric difference of two polytopes (or, geometric difference 
first, then geometric sum for minmax CLRS). In the absence of disturbances the term G(t — 1) V(t — 1) 
vanishes and no geometric difference operation is performed. Each operation of geometric sum or geo- 
metric difference for two polytopes consists in finding the vertices of the resulting polytope and calculating 
its convex hull. 

This method is implemented in the MultiParametric Toolbox (MPT) for MATLAB® [7,26]. Among 
its advantages are its simplicity, the fact that the reach sets are computed exactly, and that it is easy to 
compute the distance between two polytopes and to check whether two or more polytopes intersect, or 
whether a polytope intersects a hyperplane or a halfspace. 

However, the polytope method has its limitations. The convex hull algorithm employed by MPT is 
based on the Double description method [32] and implemented in the CDD/CDD+ package [1]. Its 
complexity is K", where K is the number of vertices and n is the state-space dimension. Hence, the use of 
MPT in general is practical only for low-dimensional systems, or for systems with very special structure 
of matrices A, B, and G that ensure that the number of polytope vertices does not grow too much with 
each time step. But even in low-dimensional systems the number of vertices in the reach set polytope 
can grow very large with the number of time steps. For example, consider the discrete-time linear time- 

cosl —sin1 
ee 1 cosl 


invariant system with A(t) = A= | B(t) =I, G(t) =0 up € {u ER? | |lulloo < 1}, and 


xo € {x ER? | ||xlloo < 1}. Starting with a rectangular initial set, the number of vertices of the reach set 


polytope is 4t + 4 at the fth step. 


29.3.4 d/dt 


In d/dt [3], the reach set of a discrete-time linear system is approximated by unions of rectangular 
polytopes [13]. The algorithm works as follows. First, given the set of initial conditions defined as a 
polytope at time fo, the evolution in time of the polytope’s extreme points is computed (11, fo, Vo). Sec- 
ond, the algorithm computes the convex hull of vertices of both, the initial polytope Vp and (11, to, Xo). 
The resulting polytope is then bloated (magnified) to include LU, <,<,, V(t to, Xo). Finally, this overap- 
proximating polytope is in its turn overapproximated by the union of rectangles. The same procedure is 
repeated for the next time interval [11,12], and the union of both rectangular approximations is taken, 
and so on. 

Rectangular polytopes are easy to represent and the number of facets grows linearly with dimension, 
but a large number of rectangles must be used to assure that the approximation is not overly conservative. 
Besides, the important part of this method is again the convex hull calculation whose implementation 
relies on the same CDD/CDD+ library. This limits the dimension of the system and time interval for 
which it is feasible to calculate the reach set. 

d/dt is implemented in C. 
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29.3.5 Zonotope Method 


Polytopes can give arbitrarily close approximations to any convex set, but the number of vertices can 
grow prohibitively large and, as shown in [14], the computation of a polytope by its convex hull becomes 
intractable for large number of vertices in high dimensions. Symmetric polytopes, called zonotopes [11], 
could be a solution. 


Definition 29.13: Zonotope 


A zonotope is a special class of polytopes of the form 


k 
z= fret ixaet 0 -1sqsif 


i=1 


wherein c and g,...,g% are vectors in R". 


Thus, a zonotope Z is compactly represented by its center c and generator vectors g1,..., g. The value 
k/n is called the order of the zonotope. 

The zonotope method for external approximation of reach sets of discrete-time linear systems was 
introduced in [17], implemented in the MATISSE package for MATLAB [6], and further discussed 
in [18]. In [18], the authors introduce computational tricks that work only for time-invariant linear 
systems. The advantage of zonotopes is that they are closed under the operations of affine transformation 
and geometric sum, hence, the reach set of a discrete-time linear system (Equations 29.1d and 29.3), with 
X and U(t), t > to, being zonotopes, is also an zonotope. Similar properties hold for the backward reach 
set. 

The problem with using zonotopes is that with every time step the order of the approximating zono- 
tope increases by k/n. This difficulty can be averted by limiting the number of generator vectors, and 
overapproximating zonotopes whose number of generator vectors exceeds this limit by lower-order zono- 
topes. This may affect the accuracy of the reach set approximation and potentially destroy the semigroup 
property that is inherently present in the zonotope method. 

Further limitations of zonotopes are that geometric difference of two zonotopes, intersections of 
zonotopes or zonotopes with hyperplanes or halfspaces, are not zonotopes. That presents a difficulty 
for the computation of reach sets for systems with disturbances and hybrid systems. Effective zonotope 
approximation algorithms for the geometric difference and intersections are needed. Currently, MATISSE 
does not provide a zonotope library in which these operations are implemented. 


29.3.6 CheckMate 


CheckMate [2] is a MATLAB toolbox that can evaluate specifications for trajectories starting from the 
set of initial (continuous) states corresponding to the parameter values at the vertices of the parameter 
set. This provides preliminary insight into whether the specifications will be true for all parameter values. 
The method of oriented rectangular polytopes for external approximation of reach sets is introduced 
in [36]. The basic idea is to construct an oriented rectangular hull of the reach set for every time step, 
whose orientation is determined by the singular value decomposition of the sample covariance matrix for 
the states reachable from the vertices of the initial polytope. The limitation of CheckMate and the method 
of oriented rectangles is that only autonomous (i.e., there is no control) systems are allowed, and only an 
external approximation of the reach set is provided. 

Currently, the development of CheckMate is discontinued. Therefore, we refer the reader to PHA Ver 
[8], the newly developed verification tool that uses Parma Polyhedra Library (PPL) [9] for its polyhedral 
computations. 
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29.3.7 Ellipsoidal Method 


All the geometric methods for reach set computation described above, namely polytopes, zonotopes, 
rectangular hulls, and oriented rectangles employ the notion of time step. At every time step a certain 
algorithm runs producing a new reach set for that time step. This can work only for discrete-time 
systems. The ellipsoidal method offers a different approach that works for continuous- and discrete-time 
linear systems with disturbances, with ellipsoidal constraints on the initial or target set, controls, and 
disturbances. 


Definition 29.14: Ellipsoid 


The ellipsoid E(q, Q) in R” with center q and shape matrix Q is the set 
E(q,Q) = {x ER" | ((x—g), Q | (x- 9g) <1}, 


wherein Q is positive definite (Q = Q! and (x, Qx) > 0 for all nonzero x € R"). 


Definition 29.15: Support Function 


The support function of a set X C R” is 


e(l | &) = sup (I, x). 
xEX 


In particular, the support function of an ellipsoid is 
(1 | €(q, Q)) = (hq) + (LQ"””. (29.43) 
We say that the ellipsoid € tightly overapproximates a given convex set 1 if there exist ] € R” such that 
o(+l| €)= p(tl| ¥) and XCE. 
We say that ellipsoid € tightly underapproximates given convex set 1 if there exist ] € R” such that 
o(+l|€)= p(tl|¥) and ECX. 


The equality p(+/ | €) = (+l | V) means that the boundaries of € and ¥ touch in directions / and —I. 

In [23], the authors introduce parametrized families of external and internal ellipsoids that tightly 
overapproximate and underapproximate the reach set and derive the differential equations that govern 
the evolution in time of the center and the shape matrices of these ellipsoids. The reach set is represented as 
the intersection of tight external and as the union of tight internal ellipsoids. In [25], this result is extended 
to the discrete-time case with special emphasis on systems with degenerate matrices A(t). In the next 
section we present the equations that describe ellipsoidal overapproximation and underapproximation 
of reach sets. 

The ellipsoidal method provides the following benefits: 


« Approximating the reach set of an n-dimensional discrete-time linear system by L ellipsoids over 
t time steps requires t[L(8n> + 4n? + 2n) + 2n?] scalar multiplications. The computational com- 
plexity grows polynomially with the system dimension, in contrast with the exponential growth of 
the polytope method complexity. 
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¢ It is possible to refine the reach set approximation as much as needed by adding more ellipsoids 
to the parameterized family. Theoretically, it is possible to exactly represent the reach set of linear 
system through both external and internal ellipsoids. 

¢ Itis possible to single out individual external and internal approximating ellipsoids that are optimal 
for a given criterion (e.g., trace, volume, diameter), or a combination of such criteria. 

¢ For systems with no disturbance, there are simple analytical expressions for control sequences that 
steer the state to a desired target. 


Ellipsoidal Toolbox (ET) for MATLAB [4] implements the reach set computations described here. 


29.3.8 Parallelotope Method 


The parallelotope* method [19] employs the idea of the ellipsoidal method to compute the reach sets of 
linear systems. The reach set is represented as the intersection of a parametrized family of tight external, 
and the union of a parametrized family of tight internal parallelotopes. The evolution equations for the 
centers and orientation matrices of both external and internal parallelotopes are provided. This method 
also finds controls that can drive the system to the boundary points of the reach set, similar to [23,37]. The 
computation to solve the evolution equations for tight approximating parallelotopes, however, is more 
involved than the one for ellipsoids, and in the case of discrete-time systems, this method does not deal 
with singular state transition matrices. 


29.3.9 Other Methods 


As was mentioned above, for certain verification problems, computation of reach sets can be avoided. For 
example, it may be enough to ensure that for given set of initial conditions %o, the trajectories of system 
(Equation 29.1) never enter a given target set }. In this case, the method of barrier certificates [34] may 
help. The idea as well as the main difficulty is to find a Lyapunov-like function C(x) such that 


1. C(x) >0iny, 
2. C(x) <0in % 
3. (DyC(x), f(t, x, u)) < 0 where C(x) = 0 


If such a function exists, system (Equation 29.1) is “safe” with respect to the initial set Yo and the target 
set )j, that is, system trajectories emanating from %% never reach )). 

Another example for which reach sets need not be computed exactly occurs when it is possible to 
ensure that for given initial set Vo there exist system trajectories that never leave Vp. The set %p is said to 
be invariant with respect to those trajectories. 

In [12], the authors show that for certain classes of discrete-time dynamical systems with disturbances 
(Equation 29.7d) and certain initial sets %, convex constraints on controls, and disturbances, for every 
disturbance there exist closed-loop control strategies that keep the state of the system inside %. 

For more information about invariant sets, we refer the reader to the survey paper [15] and references 
therein. 


29.4 Ellipsoidal Method 


Consider a continuous-time linear system 
x(t) = A(t)x(t) + B(t)u+ G(t)v, (29.44) 


in which x € R” is the state, u € R™ is the control, and v € R@ is the disturbance. A(t), B(t), and G(t) 
are continuous and take their values in R”™", R”*™, and R”*4, respectively. Control u(t, x(t)) and 


*  Parallelotope is a zonotope with n generator vectors in R”. 
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disturbance v(t) are measurable functions restricted by ellipsoidal constraints: u(t, x(t)) € E(p(t), P(t)) 
and v(t) € E(q(t), Q(t)). The set of initial states at initial time fo is assumed to be the ellipsoid E(x9, Xo). 

The reach sets for systems with disturbances computed by the ET are CLRS. Henceforth, when describ- 
ing backward reachability, reach sets refer to CLRS or CLBRS. Recall that for continuous-time linear 
systems maxmin and minmax CLRS coincide, and the same is true for maxmin and minmax CLBRS. 

If the matrix Q(-) = 0, the system (Equation 29.44) becomes an ordinary affine system with known 
v(-) = q(-). If G(-) =0, the system becomes linear. For these two cases, (Q(-) =0 or G(-) =0), the 
reach set is as given in Definition 29.1, and so the reach set will be denoted as Yc (t, to, E(xo, Xo0)) = 
X(t, to, E(xo, Xo)). 

The reach set V(t, to, E(xo,Xo)) is a symmetric compact convex set, whose center evolves in time 
according to 


Xe(t) = A(t)xe(t) + BUt)p(t) + G(t)q(t), xc(to) = xo. (29.45) 
Fix a vector Jy € R”, and consider the solution /(t) of the adjoint equation 
i(t)=—AT (M(t), M(t) = bo, (29.46) 
which is equivalent to 
I(t) = ©" (to, Hh. 


If the reach set V(t, to, E(xo, Xo)) is nonempty, there exist tight external and tight internal approximat- 
ing ellipsoids € (xc(t), X7° (t)) and E(x-(t), X, (t)), respectively, such that 


E(xe(t), Xp (t)) C X(t, to, E(xo, Xo) S E(xe(t), XO), (29.47) 


and 


p(U(t) | E(xe(t), X7 (t))) = oUt) | XC to, E(x, Xo))) = oF) | E(xc(t), X/* (0). (29.48) 


The equation for the shape matrix of the external ellipsoid is 


X(t) = A(X} (1) + X7 (DAT (f) + mi(t)X7 () + Berea 


— (x/"(t))'/?8)(t)(G()Q()GT (#))/? — (G(Q(HGT (A) 2S (EX (D)/?, (29.49) 
X} (to) = Xo» (29.50) 


in which 
(I(t), B(t)P(t)B? (t)1(t))!/? 


7) = Ty, SFO)? 


and the orthogonal matrix S)(t) (S)(t)S7 (t) = I) is determined by the equation 


((t), G(t)Q(t)GT (#)l(t)) 1? 


T 4)\1/2 974) — 
SHNGMQNE™ OIG) = TO ee 


(x;* (t))/21(£). 


In the presence of disturbance, if the reach set is empty, the matrix x(t) becomes sign indefinite. For a 
system without disturbance, the terms containing G(t) and Q(t) vanish from Equation 29.49. 
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The equation for the shape matrix of the internal ellipsoid is 


X) (tf) =AWX; () +X, WAT) + (X ()/? T1(4) (BO) PBT (t)) 1? 
+ (B(t)P(t)B" (t))'/2 T} (t)(X (t))!/? — ni(t)X; (t) — — pa" (29.51) 
X; (to) = Xo, (29.52) 


in which 


(I(t), GQ(H)GT (HI) 
(I(t), X* (Ue) 1/2 


y(t) = 


and the orthogonal matrix T;(t) is determined by the equation 


(I(t), B(t)P(t)B? (t)1(t)) 1/2 


T 1/2 = 
Ti(t)(B()P(t)B? (1) /°1(t) = (I(t), X) (t)1(t)) 1/2 


(x; (t))'/21(£). 


Similar to the external case, the terms containing G(t) and Q(t) vanish from Equation 29.51 for a system 
without disturbance. 
The point where the external and internal ellipsoids touch the boundary of the reach set is given by 


Xt (Ut) 
x*(t) = x-(t) + ——_.——_~ 
¢ $ 
(0, X (Ol)? 

The boundary points x; (t) form trajectories, which we call extremal trajectories. Due to the nonsingular 
nature of the state transition matrix P(t, to), every boundary point of the reach set belongs to an extremal 
trajectory. To follow an extremal trajectory specified by parameter Jp, the system has to start at time fp at 
initial state 


Xo Ip 


= x04 ; 29.53 
BE ay Mola? : 
In the absence of disturbances, the open-loop control 
P(t)BT (t)I(t) 
uj(t) = p(t) 4 (29.54) 


(l(t), B(t)P(t)B? (t)l(t)) 1/2” 


steers the system along the extremal trajectory defined by the vector Jo. When a disturbance is present, 
this control keeps the system on an extremal trajectory if and only if the disturbance plays against the 
control always taking its extreme values. 

Expressions 29.47 and 29.48 lead to the following fact: 


LJ Elc(t), XO) = Xt to, E(x, Xo) =] Elxc(t), XO). 


(Io,lo)=1 (Io,lo)=1 


In practice, this means that the more values of lp we use to compute X(t) and X; (t), the better will be 
our approximation. 
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Remark about Discrete-Time Systems 


For discrete-time linear system 
x(t+ 1) = A(t)x(t) + B(t)u(t, x(t)) + G(t) v(t), (29.44d) 


the equivalent of Equation 29.46 is 


(t+) = (ar) (f)l(t), Ut) =, (29.46) 


which implies nonsingular A(t).* 

For discrete-time systems, maxmin and minmax CLRS do not coincide and are computed separately. 
For maxmin CLRS, the ellipsoidal approximation (external or internal) €(x-(¢ + 1), X(t + 1)) defined by 
parameter Jy € R” is computed as tight external or internal approximating ellipsoid of 


E(A(t)x¢(t), A(X (t)A! (t)) © E(B(t) p(t), B(t)P(t)B" (t))-E(G(t) q(t), G(t)Q(t)G" ()), 


and for minmax CLRS, the ellipsoidal approximation €(x,(t + 1), X;(t + 1)) is computed as tight external 
or internal approximating ellipsoid of 


E(A(t)x¢(t), A(t)X)(t)A7 (t)) -E(G(t)q(t), G(t)Q(t)G" (t)) ® E(B(t)p(t), B(t)P(t)B" (t)) 


specified by direction /(t) that satisfies Equation 29.46d. 

For details and equations related to the discrete-time case, we refer the reader to the manual of the 
ET [4]. 

Analogous results hold for the backward reach set. 

Given the terminal time ft; and ellipsoidal target set €:y;, Y1), the CLBRS Yer (t1, t, Vi) = V(t, 5 V1), 
t < t, ifit is nonempty, is a symmetric compact convex set whose center is governed by 


yet) = Ayc(t) + B)p(t) + Gq(t), yelti) = 1. (29.55) 


Fix a vector /; € R”, and consider 
I(t) = &(t, t)"h. (29.56) 


If the backward reach set V(t, t,E(y1, Y1)) is nonempty, there exist tight external and tight internal 
approximating ellipsoids E(y,(t), Y*(@) and E(y,(t), Y; (¢)) respectively, such that 


Elyc(t), ¥, (t)) SV t El. Ya) S Elye(t), YF), (29.57) 
and 
p(U(t) | E(ye(t), Yy (£))) = e(U(t) | V(t, t, E(yo, Yo))) = e(U(t) | E(yc(t), ¥;*())). (29.58) 
The equation for the shape matrix of the external ellipsoid is 
Yi (t) =AQ)Y() + YF (AT) — mi(8) Yi () — —_ BiyP(eyB" (0) 
m(t) 
+(¥*(#)'/s((GMQGG" (1)? + (GHEHG(H) 7s WO)”, (29.59) 
Y GS; (29.60) 


* The case when A(t) is singular is described in [25]. The idea is to substitute A(t) with the nonsingular As(t) = A(t) + 
8U(t)W(t), in which U(t) and W(t) are obtained from the singular-value decomposition 


A(t) = U(f)X(t)W(E). 


The parameter § can be chosen based on the number of time steps for which the reach set must be computed and the 
required accuracy. The issue of inverting ill-conditioned matrices is also addressed in [25]. 
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in which 
(I(t), B(t)P(t)B? (£)1(t)) 1/2 


(I(t), Yj (OU) 1/2 


and the orthogonal matrix S;(t) satisfies the equation 


w(t) = 


> 


(I(t), G(t)Q(t)G7 (£)l(t)) 1/2 


(I), YF (OUD)! (yh ())/71(0). 


S)(t)(G(t)Q(t)GT (t))'/71(t) = 


The equation for the shape matrix of the internal ellipsoid is 


Y(t) =AWY, (t)+ ¥, A(t) — (¥, ())!? T1()(BO)P()BT (t))'/? 
— (B(t)P(t)B? (t))/? TF (t)(¥, (t))/? + mild), (t) + Sp MANE", (29.61) 
1 
Y, (t1) = Yi5 (29.62) 
in which 
(I(t), G(t)Q(t)GT (t)1(t)) 1/? 
(I(t), Yj (et) 1/2 


and the orthogonal matrix T;(t) is determined by the equation 


y(t) = 


> 


I(t), B(t)P(t)B? (t)1(t))!/2 


. 1/2 _— ( 
T(t)(B(t)P(t)Be (t)) /°U(t) = (I(t), ¥, ()U()) 1/2 


(Y, (#))'/21(£). 


Just as in the forward reachability case, the terms containing G(t) and Q(t) vanish from Equations 29.59 
and 29.61 in the absence of disturbances. The boundary value problems (Equations 29.55, 29.59 and 29.61) 
are converted to the initial value problems by the change of variables s = —t. 

Owing to Equations 29.57 and 29.58, 


L) €0-@).¥/ O)) =VtLeo. Y= () Ect), ¥*(). 
(lh)=1 (hyh)=1 
Remark 29.3 


In expressions 29.49, 29.51, 29.59, and 29.61, the terms 1/m)(t) and 1/n;(t) may not be well defined for 
some vectors J, because matrices B(t)P(t)B! (t) and G(t)Q(t)G! (£) may be singular. In such cases, we set 
these entire expressions to zero. 


29.5 Applications 


We illustrate the ellipsoidal approach with three applications. 


29.5.1 Steering the System to a Target 


Given system Equation 29.44, target set defined by ellipsoid €(y;, Y;), and terminal time ¢;, we want to 
find a closed-loop control that steers the system from some state yo at time fo < ft, to E(y,, Y1) at ty. 

First we compute external and internal ellipsoidal approximations £(y,(t), Y;*(t)) using Equations 
29.59 and 29.60, and E(y,(t), Y, (t)) using Equations 29.61 and 29.62, to < t < t, for different values of 
the parameter /; € R”. Ifthere exists an external ellipsoid E(y,(to), yi? (to)) such that yo ¢ E(yc(to), as (to)), 
there is no closed-loop control that can guarantee taking the system from yo at fo to a state within €(y1, Y1) 
at t;. On the other hand, if there exists an internal ellipsoid E(y-(to), Y; (to)) defined by the choice of J), 
such that yo € E(y¢(to), Y, (to)), such a control does exist. 
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We build the closed-loop control u(t, y(t)) so as to keep the system state y(t) inside, if possible, or as 
close as we can, if not, to the internal approximating ellipsoid €(y,(t), Y, ()) for to < t < t). The steps 
below describe control synthesis at time f. 


1. Compute 
v(t) = (y(t) = yelt), (YP OY" — yelt))). 


If y(t) < 1, then y(t) € E(yc(t), Y; (¢)), and the control u(t, y(t)) can be chosen arbitrarily in 
E(p(t), P(t)). For example, set u(t, y(t)) = p(t). 

2. Otherwise, if y(t) > 1, y(t) is a boundary point of ellipsoid E(y-(£), y(t) Y, (t)) corresponding to 
the direction m(t) € R", 


m(t) = (¥; (t))"" (y(t) — yel(t)). 


In order to steer the system closer to the internal approximating ellipsoid, control u(t, y(t)) must 
act in the direction —m(t). 

3. Choose u(t,y(t)) so that the vector B(t)u(t,y(t)) is a boundary point of the ellipsoid 
E(B(t)p(t), B(t)P(t)B! (t)) C R” in the direction —m(t), 


P(t)B? (t)m(t) 
(m(t), B(t)P(t)B? (t)m(t)) 1/2" 


u(t, y(t)) = p(t) 
To summarize, 


p(t), if (y(t) — yelt), (YP (0) — ye(t))) <1, 
P(t)BT (t)(Y; (t)) (y(t) — yc(t)) 


(YP (2) y(t) — ye(t)), BO)P()BT (t)(Y_ (1) (v(t) — ye(t))) 1/2’ 


u(t, y(t)) = 


otherwise. 


p(t) 


(29.63) 

The rigorous proof that this closed-loop control works can be found in [22]. In [16], the authors apply this 

technique to stop a high-dimensional oscillating system using the ET for backward reach set computation. 

Formula (Equation 29.63) holds for discrete-time linear systems, except that instead of Y, (f), shape 
matrices of internal approximating ellipsoids for maxmin or minmax CLBRS must be used. 


29.5.2 Switching System 


A switching system is a system whose dynamics changes at known times. Consider the RLC circuit shown 
in Figure 29.1. It has two inputs, the voltage v and current i. Define 


* x), the voltage across capacitor C), so C)x; is the corresponding current. 
* x2, the voltage across capacitor C2, so the corresponding current is C2x2. 
* x3, the current through the inductor L, so the voltage across the inductor is Lx3. 


FIGURE 29.1 RLC circuit with two inputs. 
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Applying Kirchoff’s laws we arrive at the linear system, 


1 1 

Ric, Gi Be. we 
x1 l x1 RiCy Cy v 
jo |= 0 0 — |lml+] 9 i |. (29.64) 
‘ oy i 
x3 x3 

1 1 Ro 0 0 

LE L L 


The parameters Rj, Ro, Ci, C2, and L, as well as the inputs, may depend on time. Suppose, for time 
0<t<2,R) =2Q, Ro = 10, C; =3 F, C) =7 F, L=2 H, and both inputs, v and i, are present and 
bounded by ellipsoid €(0,1); and for time t > 2, Ry = Rp = 2Q,C, = Cp =3 F, L=6 H, the current 
source is turned off, and |v| < 1. Then, system (Equation 29.64) can be rewritten as 


ah, “age fai Pi 

6 si Xx] 6 3 : 

0 0 5 x2I|+!]o oO He 0<t<2, 
ie A: oe SE toe Gg 
i z 2 a (29.65) 
eg eis, gs, tee 1 

6 3 X] 6 

0 0 ; x2/+}]o |» Dt 

Te ok Eee 0 

6 6 3 


We can use the ET to compute the reach set of Equation 29.65 for some time t > 2, say, t = 3. 


>> % define system 1: 
>> Al = [-1/6 0 -1/3; 0 0 1/7; 1/2 -1/2 -1/2]; 
>> Bl = [1/6 1/3; 0 0; O OJ; 


>> Ul = ellipsoid(eye(2)); 
>> sl = linsys(Al, Bl, U1); 
>> 


>> % define system 2: 
>> A2 = [-1/6 0 -1/3; 0 0 1/3; 1/6 -1/6 -1/3]; 
>> B2 = [1/6; 0; 0]; 
>> U2 = ellipsoid(1); 


>> s2 = linsys(A2, B2, U2); 

>> 

>> XO = ellipsoid(0.01*eye(3)); % set of initial states 
>> LO = eye(3); % 3 initial directions 

>> TS = 2; % time of switch 

>> T = 3; % terminal time 

>> 


>> % compute the reach set: 


>> rsl = reach(sl, X00, LO, TS); % reach set of the first system 
>> % computation of the second reach set starts 

>> %® where the first left off 

>> rs2 = evolve(rsl, T, s2); 

>> 


>> % obtain projections onto (xl, x2) subspace: 
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>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


BB = [1 0 0; 


psl = projection(rsl1, 
ps2 = projection(rs2, 


subplot(2, 2, 
plot_ea(ps1, 
hold on; 
plot_ia(ps1, 
plot_ea(ps2, 
plot_ia(ps2, 


fe) 


subplot(2, 2, 


01 


1); 


'r’'); 


'g') 
'y') 
b’) 


t 


2); 


plot_ea(cut(rs2, 


hold on; 


plot_ia(cut(rs2, 


O14 


% plot the results: 
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Figure 29.2a shows how the reach set of system (Equation 29.65) projected onto the (x;,x2) plane 
evolves in time from t = 0 to f = 3. The external reach set approximation for the first dynamics is in light 
gray, the internal approximation is in dark gray. The dynamics switches at t = 2. The external reach set 
approximation for the second dynamics is in very light gray, its internal approximation is in dark gray. 
The full three-dimensional external (very light gray) and internal (dark gray) approximations of the reach 
set at t = 3 are shown in Figure 29.2b. 

To find out where the system should start at time t = 0 in order to reach a neighborhood M of the origin 
at time f = 3, we compute the backward reach set from t = 3 tot = 0. 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


>> 


M = ellipsoid(0.01*eye(3)); 


TT = 3; % terminal time 


brs2 = reach(s2, 


M, LO, 


brsl = evolve(brs2, 0, 


2 


bpsl = projection(brs1, 


bps2 = projection(brs2, 


2 


subplot(2, 2, 


plot_ea(bps1, 
(red) 
hold on; 
plot_ia(bps1, 
(green 
plot_ea(bps2, 
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plot_ia(bps2, 
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(>) 9, 


(d) 


a) 


FIGURE 29.2 Forward and backward reach sets of the switched system (external and internal approximations). The 
dynamics switches at t = 2. (a) Forward reach set for the time interval 0 < t < 3 projected onto (x1, x2) subspace. (b) 
Forward reach set at t = 3 in R*. (c) Backward reach set evolving from t = 3 to t = 0 projected onto (x1, x2) subspace. 
(d) Backward reach set at t = 0 in R°. 


>> % plot the 3-dimensional backward reach set at time t = 0: 
>> subplot(2, 2, 4); 

>> plot_ea(cut(brs1, 0), ‘r’); 

>> hold on; 

>> plot_ia(cut(brsl, 0), ‘'g’); 


Figure 29.2c presents the evolution of the reach set projected onto the (x1,x2) plane in backward 
time. Again, external and internal approximations corresponding to the first dynamics are shown in dark 
and light gray, and these corresponding to the second dynamics in very light and dark gray. The full 
dimensional backward reach set external and internal approximations of system (Equation 29.65) at time 
t = 0 is shown in Figure 29.2d. 


29.5.3 Hybrid System 


There is no explicit implementation of the reachability analysis for hybrid systems in the ET. Nonetheless, 
the operations of intersection available in the toolbox allow us to work with certain class of hybrid sys- 
tems, namely, hybrid systems with affine continuous dynamics whose guards are ellipsoids, hyperplanes, 
halfspaces, or polytopes. 

We consider the switching-mode model of highway traffic presented in [33]. The highway segment is 
divided into N cells as shown in Figure 29.3. In this particular case, N = 4. The traffic density in cell i is 
x; vehicles per mile, i = 1, 2, 3, 4. 

Define 


¢ vj, the average speed in mph, in the ith cell, i = 1, 2, 3, 4. 
¢ wij, the backward congestion wave propagation speed in mph, in the ith highway cell, i = 1, 2, 3, 4. 
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FIGURE 29.3 Highway model. (Adapted from L. Mufioz, et al. In American Control Conference, pp. 3750-3755, 
2003.) 


* xmi, the maximum allowed density in the ith cell; when this value is reached, there is a traffic jam, 
i= 1,2,3,4. 

¢ dj, the length of ith cell in miles, i = 1, 2, 3, 4. 

¢ T;, the sampling time in hours. 

« b, the split ratio for the off-ramp. 

¢ uy, the traffic flow coming into the highway segment, in vehicles per hour (vph). 

* uy, the traffic flow coming out of the highway segment (vph). 

* u3, the on-ramp traffic flow (vph). 


Highway traffic operates in two modes: free-flow in normal operation; and congested mode, when there is 
a jam. Traffic flow in free-flow mode is described by 


Vi Ts 
| 0 0 wT 
dy ls 0 0 
T T dy 
x [t+ 1] Vits 1- V215 0 0 x, [ft] ss 
x2lt+1]] _ dy dy x(t] 4{ 0 0 aes . 
x3lt+1] | ‘i v2Ts ees Ts g xs[t] da A 
xalt+ 1] ds ds xa[t] 0 0 0 ; 
T. T. 
0 G- Arse a ee 
d4 d4 
(29.66) 
The equation for the congested mode is 
Wi Ts w2 Ts 
_ 0 0 
dy dy 0 0 wT; 
xy[t+1] 0 ,_”% Ts w3Ts 0 x1(t] qd) ? 
xolt+1)] _ dn do x(t] it 0 0 0 . 
xslt+1]|_ , ‘ 1 m3Ts 1 waTs | | ale] 0 Oo 0 i: 
x4[t + 1] a3 1-b a& xa4lt] wT; 
waTs ~~ d 0 
0 0 0 1-— 4 
d4 
WwW) Ts = w2 Ts 0 0 
dy dy 
0 mh i mh 0 XM1 
ae : 2 Pie (29.67) 
6 , w3Ts 1 waTs | | xm3 
a3 1-—b d3 XM4 
T. 
0 0 0 Wats 
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The switch from the free-flow to the congested mode occurs when the density x2 reaches x42. In other 
words, the hyperplane H([0 1 0 ol’, Xmz2) is the guard. (When the state enters the guard, the system 
equation switches.) 

We indicate how to implement the reach set computation of this hybrid system using the ET. We first 
define the two linear systems and the guard. 


>> % assign parameter values: 

>> vl = 65; v2 = 60; v3 = 63; v4 = 65; $% 

>> wl = 10; w2 = 10; w3 = 10; w4 = 10; % mph 
>> dl = 2; d2 = 3; d3 = 4; d4 = 2; % miles 
2/3600; % sampling time in hours 


>> Ts = 

>> xM1 = 200; xM2 = 200; xM3 = 200; xM4 = 200; % vehicles per lane 
>> b = 0.4; 

>> 

>> Al = [(1-(vl1*Ts/d1)) 0 0 0 


(v1l*Ts/d2 (1-(v2*Ts/d2)) 0 0 
0 (v2*Ts/d3) (1-(v3*Ts/d3)) 0 


0 0 ((1-b)*(v3*Ts/d4)) (1-(v4*Ts/d4))]; 
>> Bl = [vl*Ts/d1 0 0; 0 0 v2*Ts/d2; 00 0; 0 0 0]; 
>> Ul = ellipsoid([180; 150; 50], [100 0 0; 0 100 0; 0 0 25]); 
>> 
>> A2Q = [(1-(wl*Ts/dl1)) (w2*Ts/d1) 0 0 
0 (1-(w2*Ts/d2)) (w3*Ts/d2) 0 
0 0 (1-(w3*Ts/d3)) ((1/(1-b))* (w4*Ts/d3) ) 
000 (1-(w4*Ts/d4))]; 
>> B2 = [0 0 wl*Ts/d1l; 0 0 0; 0 0 0; O -w4*Ts/d4 0]; 
>> U2 = U1; 
>> G2 = [(wl*Ts/dl1) (-w2*Ts/dl1) 0 0 
0 (w2*Ts/d2) (-w3*Ts/d2) 0 


0 0 (w3*Ts/d3) ((-1/(1-b) )* (w4*Ts/d3) ) 

00 0 (w4*Ts/d4)]; 
>> V2 = [xM1; xM2; xM3; xM4 
>> 


>> % define linear systems: 
>> sl = linsys(Al, Bl, U1, [], [], [1], [], ‘’d’); 
>> s2 = linsys(A2, B2, U2, G2, V2, [], [], ‘'d’); 
>> 


free-flow mode 


dP dP 


congestion mode 


>> % define guard: 
>> GRD = hyperplane([0; 1; 0; O], xM2); 


We assume that initially the system is in free-flow mode. Given a set of initial conditions, we compute 
the reach set according to dynamics (Equation 29.66) for certain number of time steps. We consider an 
external approximation of the reach set by a single ellipsoid. 


>> initial conditions: 
>> XO = [170; 180; 175; 170] + 10*e11_unitball (4) ; 


>> LO = [1; 0; 0; 0]; % single initial direction 

>> N = 100; % number of time steps 

>> 

>> ffrs = reach(sl, X0, LO, N); % free-flow reach set 

>> EA = get_ea(ffrs); % 101x1l array of external ellipsoids 
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Having obtained the ellipsoidal array EA representing the reach set evolving in time, we determine the 
ellipsoids in the array that intersect the guard. 


>> I = hpintersection(EA, GRD); % some of the intersections are empty 


2 


>> D = find(~isempty(I)); % determine nonempty intersections 


>> min(D) 


69 


Analyzing the values in array D, we conclude that the free-flow reach set has nonempty intersection 
with hyperplane GRD at t = 18 for the first time, and at t = 68 for the last time. Between t = 18 and t = 68 
the reach set crosses the guard. Figure 29.4a shows the free-flow reach set projected onto the (x1, x2, x3) 
subspace for t = 10, before the guard crossing; Figure 29.4b for t = 50, during the guard crossing; and 
Figure 29.4c for t = 80, after the guard was crossed. 

For each time step that the intersection of the free-flow reach set and the guard is nonempty, we 
establish a new initial time and a set of initial conditions for the reach set computation according to 
dynamics (Equation 29.67). The initial time is the array index minus one, and the set of initial conditions 
is the intersection of the free-flow reach set with the guard. 


>> crs = []; 
>> for i = 1:size(D, 2) 
rs = reach(s2, I(D(i)), LO, [(D(i)-1) N]); 
crs = [crs rs]; 
end 


The union of reach sets in array crs forms the reach set for the congested dynamics. 

A summary of the reach set computation of the linear hybrid system (Equations 29.66 and 29.67) for 
N = 100 time steps with one guard crossing is given in Figure 29.4d, which shows the projection of the 
reach set trace onto the (x1, x2) subspace. The system starts evolving in time in free-flow mode from a set 
of initial conditions at tf = 0, whose boundary is shown in magenta. The free-flow reach set evolving from 
t = 0 to f = 100 is shown in blue. Between t = 18 and t = 68, the free-flow reach set crosses the guard. 
The guard is shown in red. For each nonempty intersection of the free-flow reach set and the guard, 
the congested mode reach set starts evolving in time until t = 100. All the congested mode reach sets 
are shown in green. Observe that in the congested mode the density x2 in the congested part decreases 
slightly, while the density x; upstream of the congested part increases. The blue set above the guard is not 
actually reached, because the state evolves according to the green region. 


29.6 Conclusion 


Control problems with hard bounds on the control set and for which finite time behavior has to meet 
guarantees (reaching or avoiding a target set) despite disturbances cannot be addressed by traditional 
methods of design. Central to recent approaches to solving these design problems are the concept and 
calculation of the OLRS or CLRS. Effective reach set computational tools have been developed over the 
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FIGURE 29.4 Reach set of the free-flow system is dark gray, reach set of the congested system is very light gray, the 
guard is gray straight line. (a) Reach set of the free-flow system at t = 10, before reaching the guard (projection onto 
(x1, X2,x3)). (b) Reach set of the free-flow system at t = 50, crossing the guard (projection onto (x1, x2, x3)). (c) Reach 
set of the free-flow system at t = 80, after the guard is crossed (projection onto (x1, x2, x3)). (d) Reach set trace from 
t =0 to t= 100, free-flow system in dark gray, congested system in green; bounds of initial conditions are marked 
with light gray (projection onto (x1, x2)). 


past decade. These tools have been described. Among these tools, the ellipsoidal approach is the most 
promising. That approach, embodied in the ET, is illustrated by three examples. 
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30.1 Introduction 


Hybrid systems are heterogeneous dynamical systems, the behaviors of which are determined by inter- 
acting continuous variables and discrete switching logics [1,2]. By heterogeneity, we mean hybrid systems 
containing two different kinds of dynamics. One is time-driven continuous variable dynamics, usually 
described by differential or difference equations; the other is event-driven discrete logic dynamics, whose 
evolutions depend on “if-then-else” type of rules and may be described by automata or Petri nets. In addi- 
tion, these two kinds of dynamics interact with each other and generate complex dynamical behaviors, 
such as switching once certain continuous variable passes through a threshold, or state jumping when 
certain discrete event occurs. As a simple example, the temperature regulation in an air-conditioned 
room can be considered as a hybrid system; the room temperature evolution forms the continuous vari- 
able dynamics following thermophysical laws, whereas the on-off evolution of the air conditioner can be 
modeled as a discrete event process. 

Hybrid systems have been identified in a wide variety of applications; in the control of mechanical sys- 
tems, in process control, in automotive industry, power systems, aircraft, and traffic control, among many 
other fields. Specifically, hybrid systems have a central role in embedded control systems, where program 
codes interact with the physical world. In particular, the logic rules programmed in the embedded devices, 
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which can be modeled as discrete event systems, are affected and influenced by the continuous variable 
physical processes, such as spatial location, temperature, and pressure evolutions. Studies in hybrid sys- 
tems could provide a unified modeling framework for embedded systems, and systematic methods for 
performance analysis, verification, and embedded microcontroller design. Therefore, hybrid systems have 
attracted the attention of researchers not only from control engineering, but also from computer science 
and mathematics. Topics, such as modeling, verification, stability, controllability, optimal control, and 
supervisory control, have been extensively studied in the hybrid system literature, and the interested 
readers may refer to [1,2,4,5,13] and the references therein. In this chapter, we focus on the stability issues 
of hybrid systems. 

It is known that the stability of hybrid systems includes several interesting phenomena due to the 
interaction of continuous variable dynamics and discrete switching logics [3,6,8]. For example, even 
when all the continuous variable subsystems are exponentially stable, the hybrid system may have 
divergent trajectories under certain discrete switching logic. On the other hand, one may carefully 
switch between unstable continuous variable subsystems to make the overall hybrid system exponen- 
tially stable. As these examples suggest, the stability of hybrid systems depends not only on the con- 
tinuous variable dynamics of each subsystem, but also on the properties of discrete switching logics. 
Therefore, the stability study of hybrid systems can be roughly divided into two kinds of problems. 
One is the stability analysis of hybrid systems under given discrete switching logics; the other is 
the synthesis of stabilizing switching logics for a given collection of continuous variable dynamical 
systems. 

We mainly focus on a subclass of hybrid systems that consists of a finite number of continuous-variable 
subsystems and a discrete logical rule, which orchestrates switching between these subsystems. The sys- 
tems are usually called switched systems in the literature [6,8]. In this chapter, we use the terms “hybrid sys- 
tems” and “switched systems” interchangeably. One convenient way to classify hybrid/switched systems is 
based on the dynamics of their subsystems, for example, continuous-time or discrete-time, linear or non- 
linear and so on. In this chapter, we focus our attention to hybrid/switched systems where all subsystems 
are linear time-invariant (LTI) systems. The generalization of these results to nonlinear switched systems 
or more general cases are well documented in the literature; see, for example, survey papers [3,6,8] for 
further references. 

The rest of this chapter is organized as follows. First, we focus on the stability analysis of hybrid systems 
under given discrete switching logics. In particular, some results on the stability analysis for hybrid systems 
under arbitrary switching are introduced in Section 30.2, while the stability under slow switching (like 
dwell time and average dwell time) is studied in Section 30.3.1. The general case of hybrid system stability 
under restricted switching is investigated in Section 30.3 through multiple Lyapunov functions. Then, we 
turn to the synthesis of stabilizing switching logic for a given collection of continuous variable dynamical 
systems in Section 30.4, where several stabilization conditions and design methods are described. Finally, 
the chapter concludes with remarks and a list of references. 


30.2 Arbitrary Switching 


In this section, we first consider the stability analysis problem where there are no restrictions on the 
discrete event dynamics of the hybrid system. This may be due to our lack of knowledge of the discrete 
event logic, or of the partitions of the state space, or of the constraints in the hybrid system of concern. 
Under these circumstances, one usually tends to be more conservative and assumes that all discrete 
switchings are possible; this is called arbitrary switching in the literature. On the other hand, when the 
stability under arbitrary switching is guaranteed, this could provide us with flexibility in the discrete logic 
design, where one may focus on improving the performances, since the closed-loop system stability is not 
a problem any longer. 
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30.2.1 Common Lyapunov Function 


We know that a hybrid system may become unstable even when all subsystems are exponentially stable. 
Therefore, to identify conditions under which a hybrid system is stable under arbitrary switching is 
nontrivial and interesting. For this, it is necessary that all the subsystems are asymptotically stable, since if 
one subsystem were unstable, one switching strategy would have been to always select that subsystem all 
the time, which is a valid “switching logic” as well, and that would make the system unstable. In general, 
the above subsystems’ stability assumption is not sufficient to ensure stability for the hybrid systems 
under arbitrary switching. However, if there exists a common Lyapunov function for all the subsystems, 
that is, a continuously differentiable, radially unbounded, positive definite function V : R x R” > R, for 
which the derivative V(x, t) is negative definite along all subsystems’ trajectories, then the stability of the 
hybrid systems is guaranteed under arbitrary switchings. This provides us with a possible way to solve 
this problem, and a lot of research efforts have been focused on finding common quadratic Lyapunov 
functions (CQLFs). 


30.2.1.1 Common Quadratic Lyapunov Functions 


First, we consider a collection of continuous-time LTI systems 
(t)=Ajx(t), teR*t, ie3, (30.1) 


where 3 stands for a finite index set. For all i € 3, the state matrices Aj € R”*”. Note that the origin x. = 0 
is a common equilibrium for the systems described in Equation 30.1. The hybrid system of interest is 
built by allowing arbitrary switching among these LTI systems (Equation 30.1). 

A CQIF for Equation 30.1 is a special class of Lyapunov functions of the form 


V(x) =x! Px, (30.2) 


where P = P? (symmetric) and P > 0 (positive definite). In addition, its time derivative along any trajec- 
tory of systems (Equation 30.1) is negative definite, or alternatively 


A} P+PA;=—-Q; i€3, (30.3) 


where Q; are symmetric and positive definite for all i ¢ 3. The existence of a CQLF for all its subsystems 
assures the quadratic stability of the hybrid system. Quadratic stability is a special class of exponential 
stability, which implies asymptotic stability, and has attracted a lot of research efforts due to its importance 
in practice. 

A CQIF for Equation 30.1 can be obtained by solving a set of linear matrix inequalities (LMIs). for 
example, there exists a positive definite symmetric matrix P, P € R"*", such that 


PA;+A}P <0, VieT, (30.4) 


hold simultaneously. However, the standard interior point methods for LMIs may become ineffective as 
the number of subsystems increases. This motivates researchers to identify easily verifiable conditions 
that guarantee the existence of a CQLF for Equation 30.1. Here, we take a look at a well-studied special 
case in the literature; interested readers may refer to [6,8] for further references. 


30.2.1.2 Commutative Systems 


Let us first look at a special case, where the subsystems’ state matrices are pairwise commutative, that is, 
AjAj = AjAj for all i,j <¢ 3. Because of the commutativity, it is easy to show that 


ky yko _ gho yky 
A; 4; =A; A; 
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for any nonnegative integer k, and kz, and 


eAitt efit = eit pAiti 


for any nonnegative real number ¢; and ty. By direct computation, it is straightforward to verify that in 
this case the arbitrary switching system is stable if and only if all its subsystems are stable. 


Theorem 30.1: 


For a collection of LTI systems (Equation 30.1) with the index set J = {1,...,N}, if all subsystem matrices 
are stable (i.e., all eigenvalues of A; have negative real part) and commute pairwise (AjAj = AjAj, Vi,j € 3), 
then the hybrid system with subsystems given by Equation 30.1 is asymptotically stable under arbitrary 
switching. 


ACQLE exists in this case, and can be determined by solving a collection of chained Lyapunov equations 
as shown in the following: 


Theorem 30.2: 


Assume that the index set J = {1,...,N}. Let P},..., Py be the unique symmetric positive definite matrices 
that satisfy the Lyapunov equations 


ATP, +P\A; =I, (30.5) 
ATP; +P;Aj=—P;-1, i=2,...,N. (30.6) 


Then the function V(x) = x! Pyx isa CQLF for systems x(t) = Ajx(t),i=1,...,N. 
In addition, the matrix Py can be expressed in integral form as 


lo) T CO T 
Py =[ enn... (/ Aner) . + @ANN diy, 
0 0 


It is not difficult to extend this result to the discrete-time case. 


Theorem 30.3: 
Let P\,...,Pn be the unique symmetric positive definite matrices that satisfy the Lyapunov equations 
ATP|A, +P; =—I, (30.7) 
Al P;A; +P; =—P;-1, i=2,...,N. (30.8) 


Then the function V(x) = x! Pyx is a common Lyapunov function for the systems x[k + 1] = A;x[k], 
PH=A,..i5NG 


In the literature, there exist several interesting necessary and also sufficient algebraic conditions for 
the existence of a CQLF for more general cases but usually for low-dimensional systems, and interested 
readers may consult [6,8] for further references. Note that Lie algebraic conditions were proposed in the 
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literature; see, for example, [6], for arbitrary switching systems, which are based on the solvability of the 
Lie algebra generated by the subsystems’ state matrices. It was shown that if the Lie algebra generated by 
the set of state matrices A; is solvable, then there exists a CQLF, and the hybrid system is stable under 
arbitrary switching. 


30.2.2 Switched Quadratic Lyapunov Functions 


It should be pointed out that the existence of a CQLF is only sufficient for the stability of arbitrary 
switching systems. Therefore, in general, the existence of a CQLF is only sufficient for the asymptotic 
or exponential stability of hybrid systems under arbitrary switching, and could be rather conservative. 
Hence, some attention has been paid to a less conservative class of Lyapunov functions, called switched 
quadratic Lyapunov functions. 
In this subsection, we investigate the stability of the following discrete-time arbitrary switching LTI 
systems: 
x{k+1)=Ajx(k], teZ*, (30.9) 


where x € R", andi € J. Basically, since every subsystem is stable, there exists a positive definite symmetric 
matrix P; that solves the Lyapunov equation for each ith subsystem: 


Al P;Aj — P; <0, 


for all i € Z. Next, these matrices, P;, are patched together based on the switching signals to construct a 
global Lyapunov function as 
V(k, x[k]) = x" [k1Pocayx1k], (30.10) 


where o(k) : k — Z stands for the switching signal at step k. Since all P; are positive definite, it is clear that 
the function V(k, x[k]) = x? [k] Pocky) x1k] is also positive definite. If it further holds that AV(k, x[k]) = 
V(k+1,x[k+ 1]) — V(k, x[k]) is negative definite along the solution of Equation 30.9, then the origin of 
the system (Equation 30.9) is globally asymptotically stable. In particular, a sufficient condition for the 
stability of the arbitrary switching system (Equation 30.9) is given as follows. 


Theorem 30.4: 


If there exist positive definite symmetric matrices P; € R"*" (P; = P!) fori € T, satisfying 


Py. cAPP; 
Ee P, >0 (30.11) 


for alli,j € Z, then the linear system (Equation 30.9) with arbitrary switching is asymptotically stable. 


The stability checking for arbitrary switching linear systems can be performed by solving LMIs. 

It is clear that when P; = P; for all i,j ¢ Z, the switched quadratic Lyapunov function becomes the 
CQLF. Therefore, the stability criteria based on the switched quadratic Lyapunov function generalize 
the CQLF approach and usually give us less conservative results. However, it is worth pointing out that 
the switched quadratic Lyapunov function method is still only a sufficient condition. 


30.2.3 Necessary and Sufficient Conditions 


In the sequel, we will provide some necessary and sufficient conditions for the asymptotic stability of 
arbitrary switching linear systems. It is shown that the asymptotic stability problem for hybrid linear 
systems with arbitrary switching is equivalent to the robust asymptotic stability problem for polytopic 
uncertain linear time-variant systems, for which several strong stability conditions exist. 
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Theorem 30.5: [8,10] 


The following statements are equivalent: 


1. The arbitrary switching system 
X(t) = Ag(ryx(t), 


where A(t) € {A}, A2,..., An}, is asymptotically stable; 
2. The linear time-variant system 
x(t) = A(t)x(t), 


where A(t) € A=Conv{Aj,A2,...,An}, where Conv{-} stands for the convex combination, is 
asymptotically stable; 

3. There exists a full column rank matrix L € R™*", m > n, anda family of matrices {Aj € R™*" : i ¢ T} 
with strictly negative row dominating diagonal, that is, for each A;,i € T its elements satisfying 


akk+ > lanl <0, k=1,...,m, 
k#l 


such that the matrix relations 


are satisfied. 


It is interesting to note that the nice property of A; (i¢Z) implies the existence of a CQLF for 
the higher-dimensional arbitrary switching system. Unfortunately, applying the above Theorem is still 
difficult, because, in general, the numerical search for the matrix L is not simple. However, this equivalence 
bridges two research fields, namely the fields of hybrid system and robust stability. Therefore, existing 
results in the robust stability area, which has been extensively studied for over two decades, can be 
directly introduced to study the arbitrarily switching systems and vice versa. For example, it is known 
in the robust stability literature that the global attractiveness, (global) asymptotic stability, and (global) 
exponential stability are all equivalent for the polytopic uncertain linear time-variant systems [10]. Hence, 
these stability concepts are also equivalent for arbitrary switching systems. Similar results can be developed 
for the discrete-time case as shown below. 


Theorem 30.6: [8,10] 


The following statements are equivalent: 


1. The arbitrary switching system x[k +1] = Agcg)x[k], where Ag) € {A1, A2,..., An}, is asymptoti- 
cally stable. 

2. The linear time-variant system x[k + 1] = A(k)x[k], where A(k) € A= Conv{A}, A2,...,An}, is 
robustly asymptotically stable. 

3. There exists an integer m>n and Le R"*™, rank(L) =n such that for all Aj,i€ TZ, there exists 
A; € R™*™ with the following properties: 
a ALL SAF, 
b. Each column of A; has no more than n nonzero elements and 


m 
Aj = max ani| <1. 
IAilloo ee | 
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Based on the equivalence between the asymptotic stability of arbitrary switching linear systems and 
the robust stability of polytopic uncertain linear time-variant systems, some well-established converse 
Lyapunov theorems can be introduced for arbitrary switching linear systems. For example, the following 
results were taken from [10]. 


Theorem 30.7: 


If the arbitrary switching system is exponentially stable, then it has a strictly convex, homogenous (of second 
order) common Lyapunov function of a quasi-quadratic form V(x) = x! L(x)x, where L(x) = L? (x) = 
L(tx) for all nonzero x € R" andt eR. 


Furthermore, we may restrict our search to include only polyhedral Lyapunov functions (also known 
as piecewise linear Lyapunov functions) as the following result pointed out. 


Theorem 30.8: 


If an arbitrary switching linear system is asymptotically stable, then there exists a polyhedral Lyapunov 
function, which is monotonically decreasing along the switched system’s trajectories. 


This converse Lyapunov theorem holds for both discrete-time and continuous-time cases, which 
suggests that the existence of a common Lyapunov function (not necessarily quadratic) is not only 
sufficient, but also necessary for the stability of a hybrid system under arbitrary switching. 

Before we move on to another topic, let us take a look at the following example, which is taken from 
the robust stability literature. 


Example 30.1: 


Consider an arbitrary switching system, x = Ajx,i € {1,2}, where 


0 1 0 1 
ee a =| Poa ah 


It is known that no CQLF exits. However, the arbitrary switching system is asymptotically stable, 
which is assured by the existence of a piecewise quadratic Lyapunov function; a particular piecewise 
linear Lyapunov function is also suggested in the robust literature. 


30.3 Restricted Switching 


Hybrid systems may fail to preserve stability under arbitrary switching. On the other hand, one may have 
some knowledge about the occurrence of possible discrete event dynamics in the hybrid systems and 
this knowledge can be translated into restrictions on the switching signals. For example, there must exist 
certain bounds on the time interval between two successive switchings, which may be due to the fact that 
the state trajectories have to spend some finite period of time in traveling from the initial set to certain 
boundary sets before switching, if these two sets are separated. With such kind of prior knowledge about 
the switching signals, we may derive stronger results on the stability for a given hybrid system instead of 
just using the worst-case arguments of the previous section. 
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30.3.1 Slow Switching 


By studying the cases where divergent trajectories are generated through switching between two stable 
systems, one may note that the unboundedness is caused by the failure to absorb the energy increase 
caused by frequent switchings. In addition, when there is an unstable subsystem present (e.g., controller 
failure or sensor fault), if one either stays too long on it or switches too frequently to it, this may cause 
instability. Therefore, a natural question is what if we restrict the switching signals to some constrained 
subclasses. Intuitively, if one stays at stable subsystems long enough and switches less frequently, that is, 
slow switching, one may trade off the energy increase caused by switching or unstable modes, and it should 
perhaps become possible to attain stability. These ideas are proved to be reasonable and are captured by 
concepts such as dwell time and average dwell time [4] between switchings that are introduced below. 

The simplest way to characterize the concept of slow switching is perhaps to request a lower bound on 
two consecutive switching times. 


Definition 30.1: 


A positive scalar tq is called the dwell time if the time interval between any two consecutive switchings is no 
smaller than tq. 


Assume that all subsystems of the hybrid system are exponentially stable. Then, it can be shown that 
there exists a scalar tg > 0 such that the hybrid system remains exponentially stable if the dwell time is 
larger than ty. In addition, one may give an estimate on the bound of the dwell time and decay rate. 

In fact, it really does not matter if one occasionally has a smaller dwell time between switching, provided 
this does not occur too frequently. This concept is captured by the concept of “average dwell-time.” 


Definition 30.2: 


A positive constant Tq is called the average dwell time if No(t) < No +t/ta holds for all t > 0 and some 
scalar No > 0, where No(t) denotes the number of discontinuities of a given switching signal o over [0,t). 


Here the constant t, is called the average dwell time and No the chatter bound. The reason for calling 
a class of switching signals that satisfy 
t 
No(t) < No+ — 
Ta 
having an average dwell no less than tg is because 


t t 
No(t) < No+ > = Tq. 
ol) S : Ta No(t)—No ~ i 


This means that on an average the “dwell time” between any two consecutive switchings is no smaller 


than t,. The idea is that there may exist consecutive switching separated by less than tg, but the average 
time interval between consecutive switching is not less than tg. 


Theorem 30.9: 


Assume that all subsystems, x = Ajx fori € J, in the hybrid system are exponentially stable. Then, there exists 
a scalar tq > 0 such that the hybrid system is exponentially stable if the average dwell time is larger than Tq. 


Moreover, we can also obtain a bound on the decay rate. 
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Theorem 30.10: 


Given a positive scalar 9 such that Aj + dol is stable for alli € 3. Then, for any given  € (0, do), there 
exists a finite constant tq such that the hybrid system is exponentially stable with decay rate \ provided that 
the average dwell time is no less than tq. 


The stability results for slow switching can be extended to discrete-time case, where the dwell time tg 
or average dwell time t, are counted as the number of sampling periods. In particular, 


Definition 30.3: 


A positive constant tq is called the average dwell time if Ns(k) < No +k/tg holds for all k > 0 and some 
scalar No > 0, where No(k) denotes the number of switchings of a given switching signal o over [0,k). 


Theorem 30.11: 


Given a positive scalar ko such that Aj/do is stable for all i € 3. Then, for any given d € (Xo, 1), there 
exists a finite constant tq such that the hybrid system, consisting of x[k + 1] = Ajx[k] as its subsystems, is 
exponentially stable with decay rate d provided that the average dwell time is no less than Tq. 


Interested readers may refer to the survey papers [4,6,8] for further references on the stability of hybrid 
systems under slow switchings. 

We continue our study of the stability of hybrid systems under restricted switchings in this section. 
It should be pointed out that not all restrictions on switching signals can be captured by the dwell time 
or average dwell time. For example, it is difficult to transform the invariant set constraints, guard set 
constraints, and so on, which determine the switching signals, into only dwell-time or average dwell time 
restrictions on switching signals. The main difficulty comes from the fact that most constraints in hybrid 
systems are state dependent and in the form of partitions of the state space, and so it is hard to transform 
them into pure time-dependent constraints such as dwell time, and so on. This calls for a more general tool 
to study hybrid system stability, and we introduce a powerful tool, multiple Lyapunov functions (MLFs). 


30.3.2 Multiple Lyapunov Functions 


The stability analysis under constrained switching has usually been pursued in the framework of MLFs. 
The basic idea is to use multiple Lyapunov or Lyapunov-like functions, each of which may correspond to 
a single subsystem or certain regions in the state space, concatenated together to produce a nontraditional 
Lyapunov function. The nontraditionality is in the sense that the MLF may not be monotonically decreas- 
ing along the state trajectories, may have discontinuities, and be piecewise differentiable. The reason for 
considering nontraditional Lyapunov functions is that the traditional Lyapunov function may not exist 
for hybrid systems with restricted switching signals. For such cases, one may still construct a collection 
of Lyapunov-like functions, which only requires nonpositive Lie-derivative for certain subsystems in a 
certain region of the state space instead of globally negativity conditions. 

Lyapunov-like functions are defined as a family of real-valued functions {Vj,i = 1,...,N} with certain 
properties, each associated with the vector field x = f;(x) that represents the continuous dynamics for the 
hybrid system under the ith discrete mode. 
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Definition 30.4: Lyapunov-Like Function 


By saying that a subsystem has an associated Lyapunov-like function V; in region Q; C IR", we mean that 


1. There exist constant scalars Bj > a; > 0, such that 


aillxll? < Vil) < Bill|l? 


holds for any x € Qi. 
2. For allx € Q; and x 4 0, V;(x) <0. 


Here V;(x) = (OV;(x) /Ox)fi(x). The first condition implies positiveness and radius unboundedness for 
Vi(x) when x € Q;, while the second condition guarantees the decrease of the abstracted energy, value of 
function V;(x), along the trajectories of subsystem i inside Q;. 

Suppose that these regions (2; cover the whole state space, and so a cluster of Lyapunov-like functions is 
obtained. By concatenating these Lyapunov-like functions together, we obtain a nontraditional Lyapunov 
function, called MLF, which can be used to study the global stability of hybrid systems. MLFs are proved 
to be a powerful tool for studying the stability of switched systems and hybrid systems; see, for example, 
[3,6,8,9]. There are several versions of MLF results in the literature. A very intuitive MLF result [3] is 
illustrated in Figure 30.1, where the Lyapunov-like function decreases when the corresponding mode is 
active and does not increase its value at each switching instant. Formally, this result can be stated by the 
following theorem [3]. 


Theorem 30.12: 


Suppose that each subsystem has an associated Lyapunov-like function V; in its active region Q;, each with 
equilibrium point x = 0. Also, suppose that |); Q; = R”. Let o(t) be a class of piecewise-constant switching 


FIGURE 30.1 The hybrid system is asymptotically stable if the Lyapunov-like functions’ values at the switching 
instants form a decreasing sequence. 
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sequences such that o(t) can take value i only if x(t) € Qi, and in addition 
Vj (x(tij)) < Vilx(tij)), 


where tj,; denotes the time when switched system switches from subsystem i to subsystems j, that is, x(t) € 
Q; while x(t; ;) € Qj. Then, the switched linear system (Equation 30.1) is exponentially stable under the 
switching signals o(t). 


The above MLF theorem requires that at each switching instant the Lyapunov-like function does 
not increase its value, which is quite conservative. Actually, one may obtain less conservative results. 
For example, the switching signals may be restricted in such a way that at every time when we exit 
(switch from) a certain subsystem its corresponding Lyapunov-like function value is smaller than its 
value at the previous exiting time. Then the switched system is asymptotically stable. In other words, 
for each subsystem, the corresponding Lyapunov-like function values at every exiting instant form a 
monotonically decreasing sequence. Alternatively, the decreasing tendency is captured by the Lyapunov- 
like function’s value at the entering instant instead. This case is illustrated in Figure 30.2. This result can 
be presented as follows. 


Theorem 30.13: [3] 
Assume that there exists a family of Lyapunov functions {Vj : i € 3} for each stable subsystem. If for any 
two switching instants t; and t; such that i < j and o(t;) = o(t;) we have 

Voce) (ti+1)) — Vor (x(ti+1)) S —pllx(tizi)II*, 


for some constant p > 0, then the switched system is asymptotically stable. 


Furthermore, as shown in [9], the Lyapunov-like function may increase its value during a time interval, 
only if the increment is bounded by certain kind of continuous functions as illustrated in Figure 30.3. 


FIGURE 30.2 For every subsystem, its Lyapunov-like function’s value Vj at the start point of each interval exceeds 
the value at the start point of the next interval on which the ith subsystem is activated, then the hybrid system is 
asymptotically stable. 
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FIGURE 30.3 The hybrid system can remain stable even when the Lyapunov-like function increases its value during 
a certain period. 


Interested readers may refer to the survey papers [3,6,9] and references therein. Note that all the arguments 
for continuous-time hybrid/switched systems can be extended to the discrete-time case without essential 
differences. 


30.3.3 Piecewise Quadratic Lyapunov Functions 


The critical challenge of applying the MLF theorems to practical switched/hybrid systems is how to 
construct a proper family of Lyapunov-like functions. Usually this is a hard problem. However, if one 
focuses on the linear case, piecewise quadratic Lyapunov-like functions could be attractive candidates, 
since the stability conditions in the MLF theorems can be formulated as LMIs [3,5], for which efficient 
software solution packages exist. 

Considering the hybrid system with an LTI subsystem, x(t) = A;x(f), since we do not assume that the 
subsystem is stable, there may not exist a quadratic Lyapunov function in a classical sense. However, it 
is still possible to restrict our search to certain regions of the state space, say Q; C R”, and the energy 
of the ith subsystem could be decreasing along the trajectories inside this region (there is no decreasing 
requirements outside Q2;). Suppose that the union of all these regions Q; covers the whole state space; 
then we obtain a cluster of Lyapunov-like functions. Broadly speaking, the problem entails searching for 
Lyapunov-like functions whose associated (2-region covers the state space. 

Assume that the state space R” has a partition given by {Q),..., Qu}, and these regions (2; are defined a 
prioriasa restriction of the possible switching signals (state-dependent). In this subsection, we present LMI 
conditions for the existence of quadratic Lyapunov-like functions of the form of V;(x) = x! Px, assigned 
to each region Q;. A Lyapunov-like function V;(x) = x7 P;x needs to satisfy the following two conditions: 


Condition 30.1: 


There exist constant scalars B; > oj > 0 such that 
2 2 
ajllxll" < Vilx) < Billxll 


hold for all x € Qi). 
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Consider a quadratic Lyapunov-like function candidate, V;(x) =x" P;x, and require that 
ajx! Ix < x! Pix < Bix! Ix 


holds for any x € Qj. That is 
x! (a;I — P;)x <0 
x! (P; _ BiI)x <0 
holds for all x € Qi. 


Condition 30.2: 


For all x € Q; and x #0, Vi(x) <0. 
This negativeness of the Lyapunov-like function’s derivative along the trajectories of a subsystem can be 
represented as AP;, (Pi = P?) such that 


x" [A] Pi + PAj]x <0 (30.12) 


forx€ Qi. 


30.3.3.1 Switching Condition 


In addition, based on the MLF theorem of [3], for stability it is also required that the Lyapunov-like 
functions’ values at switching instant are nonincreasing, which can be expressed by 


x" Pix < x! Pix 
for x € Qi; CQ; (] Q;. The region Q;; stands for the states where the trajectory passes from region Q; 
to Qj. 
Note that all the above matrix inequalities are constrained in a local region, such as x € Q; or Qj. A 
technique called S-procedure can be applied to replace a constrained matrix inequality condition by a 
condition without constraints. To employ the S-procedure, the regions Q; and (2; ; need to be expressed 


by or be contained in regions characterized by quadratic forms. For simplicity, we assume here that each 
region (2; has a quadratic representation or approximation, that is 


Q5 = {x| x" Qix = 0}, 
and regions 2;,; can be expressed or approximated by 
235 = {al x7 Qujx = 0}. 


Then the above matrix inequalities can be transformed into unconstrained ones based also on the 
S-procedure, namely 


Theorem 30.14: 


The system (Equation 30.1) is (exponentially) stable if there exist matrices P;(P; = P?) and scalars a > 
0,B > 0, uj; = 0, v; = 0,9; = 0, and nj; = 0, such that 


al + piQ; < Pj) < BI — viQi 
ATP; +P)Aj +0;Q; < —I (30.13) 
Pj) + nijQj < Pi 


are satisfied. 
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The above theorem is an adaptation of a result in [3]. If there is a solution to the above LMI problem, 
the exponential stability is verified. In addition, a bound on the convergence rate can be estimated: 


B 1; 
Ilx(t)|| aah 8 ||xoll, 


where x(t) is the continuous trajectory with an initial state x9, and the constants a, B are solutions of the 
LMI (Equation 30.13). Based on similar arguments, LMI- based sufficient conditions for the discrete-time 
case can be derived; see, for example, [8]. 

An example is now presented to illustrate Theorem 30.14. 


Example 30.2: 
Consider a hybrid system 


x(t) = |x. if x(t) € Q1 = {x|x"Qyx > 0}, 
(30.14) 


xt) = i z 5| x(t), if x(t) € &2 = [xIxT px > 0}, 


—0.25 —025 0.25 0.25 
Wile ie Ee 2 eng 2a ke 2 


ify that Q; (JQ; =R?. 
Solving the LMI problem in Theorem 30.14 results in a solution 


I Since Q; = —Q3, it is straightforward to ver- 


p, _[ 01000 -0.4500] ,, _ [4.3792 3.8292 
1=1_94500 41.1167]’ (2 |3.8292 6.8833 


with a value of 8 = 41.12. Hence, the hybrid system is exponentially stable. Interested readers may 
refer to [3] for details and illustration of trajectories and Lyapunov-level curves. 


Note that the above conditions are all based on MLF theorems; hence the results developed in this sub- 
section are only sufficient. To reduce the possible conservativeness, a new kind of polynomial Lyapunov 
functions has been introduced and investigated for the stability analysis of hybrid systems. The compu- 
tation of such polynomial Lyapunov functions can be efficiently performed using convex optimization, 
based on the sum of squares (SOS) decomposition of multivariate polynomials. It is also possible to use 
SOS techniques together with the S-procedure to construct piecewise polynomial Lyapunov functions, 
with each polynomial as a SOS while incorporating the state constraints, and hence to generalize piecewise 
quadratic Lyapunov functions. Interested readers may refer to the survey paper [8] for further references. 


30.4 Switching Stabilization 


Implicitly, the above MLF results provide methodologies for the design of switching logics between vector 
fields so as to achieve a stable trajectory, since MLF results characterize the conditions on switching signals, 
under which the hybrid system is stable. In this section, we explicitly consider the design of stabilizing 
switching logics for hybrid systems. The formulation of the problem can be stated as follows. 


¢ Given a collection of LTI systems x = Ajx, design switching logics so that the induced hybrid system 
is stable. 


This is usually called the switching stabilization problem in the literature. It is known that even when all 
subsystems are unstable, there still may exist stabilizing switching signals. 
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30.4.1 Quadratic Switching Stabilization 


In the switching stabilization literature, most of the work has focused on quadratic stabilization for certain 
classes of systems. A hybrid system is called quadratically stabilizable when there exist switching signals 
that stabilize the system along a quadratic Lyapunov function, V(x) = x7 Px. 

It is known that a necessary and sufficient condition for a pair of LTI systems to be quadratically 
stabilizable is the existence of a stable convex combination of the two subsystems’ matrices. Specifically, 


Theorem 30.15: 


A hybrid system that contains two LTI subsystems, x(t) = A;x(t), i = 1, 2, is quadratically stabilizable if and 
only if the matrix pencil yy(A1,A2) = {Aq | Ag = ®A1 + (1 — @)A2, 0 < @ < 1} contains a stable matrix. 


A generalization to more than two LTI subsystems was suggested by using a “min-projection strategy,” 
that is, 
o(t) = arg min x(t)! PA;x(t). (30.15) 
1€ 


Theorem 30.16: 


If there exist constants a; € [0,1], and )° <7 &i = 1 such that 


Ag = >, aA, 


ieL 


is stable, then the min-projection strategy (Equation 30.15) quadratically stabilizes the switched system. 


However, the existence of a stable convex combination matrix Ay is only sufficient for switched 
LTI systems with more than two subsystems. There are example systems for which no stable convex 
combination state matrix exists, yet the system is quadratically stabilizable using certain switching signals. 
A necessary and sufficient condition for the quadratic stabilizability of switched controller systems is as 
follows. 


Theorem 30.17: [12] 


The switched system is quadratically stabilizable if and only if there exists a positive definite real symmetric 
matrix P = P™ > 0 such that the set of matrices {A;P + PA} } is strictly complete, that is, for any x € R"/{0}, 
there exists i € I such that x" (AjP + PA! )x < 0. In addition, a stabilizing switching signal can be selected 
as o(t) = min,{x! (t)(AjP + PAT)x(t)}. 


Analogously, for the discrete-time case, it is necessary and sufficient for quadratic stabilizability to 
check whether there exists a positive symmetric matrix P such that the set of matrices {A} PA; — P} is 
strictly complete. Obviously, the existence of a convex combination of state matrices Ay automatically 
satisfies the above strict completeness conditions due to convexity, while the inverse statement is not true 
in general. Unfortunately, checking the strict completeness of a set of matrices is NP hard [12]. Interested 
readers may refer to survey papers [3,6,8] for further references. 

Quadratic stability means that there exists a positive constant € such that V(x) < —ex! x. All of these 
methods guarantee stability by using a common quadratic Lyapunov function, which is conservative in the 
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sense that there are switched systems that can be asymptotically (or exponentially) stabilized in case when 
a CQLF function does not exist. Therefore, we will turn our attention to multiple Lyapunov functions, 
and describe constructive synthesis methods based on the piecewise quadratic Lyapunov function in the 
next section, which are mainly based on [11]. 


30.4.2 Piecewise Quadratic Switching Stabilization 


According to Theorem 30.14, if there exist real matrices P; (P; = pr )and scalars a > 0,8 > 0, Wj; > 0,0; > 
0, 0; > 0, and nij = 9, satisfying 


al + jQ; < Pj < BI — vjQ; 
ATP; +P/A+0jQ<-I , 
Pi + yi jQij < Pi 


then the switched linear system (Equation 30.1) is exponentially stable. 

In contrast to the stability analysis problem, here the state-space partitions Q; are not given a priori 
any more. Actually, the state partitions Q;, which induce the state-dependent switching signals, are 
to be designed. Moreover, the state space cannot be partitioned in an arbitrary way. The partition of 
the state space should facilitate the search of proper quadratic Lyapunov-like functions, and satisfy the 
nonincreasing conditions when switching occurs. This will be discussed in detail in the following. 


30.4.2.1 State-Space Partition 


Once again, the purpose of dividing the whole state space R” into pieces, denoted as 2;, is to facilitate 
the search for Lyapunov-like functions for one of these subsystems. After successfully obtaining these 
Lyapunov-like functions associated with each region Q;, one may patch them together, following the 
conditions of the above MLF theorem, so as to guarantee global stability. 

For this purpose, it is necessary that these regions (2; cover the whole state space, that is, the following 


covering property holds: 
| Je L)---LJan=R". 


This condition merely says that there are no regions in the state space where none of the subsystems 
are activated. 

Since we will restrict our attention to quadratic Lyapunov-like functions for the purpose of computa- 
tional efficiency, we will consider regions given (or approximated) by quadratic forms 


Q; = {x € R"| x7 Qix > 0}, 


where Q; € R”*” are symmetric matrices, and i € {1,..., N}. 
The following lemma gives a sufficient condition for the covering property. 


Lemma 30.1: [11] 


If for every x € R" 


N 
>> 67 Qix = 0, (30.16) 
i=1 


where 0; > 0,1 € Z, then Ge Q; = R". 
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30.4.2.2 Switching Condition 
In order to guarantee exponential stability we also need to make sure that 


1. Subsystem i is active only when x(t) € Q). 
2. When switching occurs, it is required to guarantee that the Lyapunov-like function values do not 
increase. 


To verify the first requirement, we consider the largest region function strategy, that is, 


otx(t)) = arg (maxx Qa). (30.17) 


This is due to the selection of subsystems (at state x(t)) corresponding to the largest value of the region 
function x(t)! Q;x(t). 
Suppose that the covering condition (Equation 30.16) holds, that is, 


N 
> 0;x! Qix >0 
i=1 


for some 0; > 0,i € Z. Then, based on the largest region function strategy, the state x with the current 
active mode i satisfies x! Q;x > 0. This implies that x € Q;. So the first condition holds for the largest 
region function strategy (Equation 30.17). 

To satisfy the second energy decreasing condition at switching instants, we need to know in which 
direction the state trajectory x(t) passes through the switching surfaces. However, the switching surface 
is to be designed, and so such information is lacking in general. Then we make a compromise and require 
that 

x! Pix = x" Pix 
for states at the switching plane, that is, x € Q;M Q;. Assume that the set 2; 1 Q; can be represented by 
the following quadratic form: 
Q5N Qj = (xIx"(Q — Q)x = 0}. 
Again, applying the S-procedure, we obtain 
Pj — Pj + nij(Qi — Q) =0, 


for an unknown scalar y;,;, as the switching condition. 


30.4.2.3 Synthesis Condition 


The above discussion can be summarized by the following sufficient conditions for the collection of 
continuous-time systems (Equation 30.1) to be exponentially stabilized. 


Theorem 30.18: [11] 


If there exist real matrices P; (Pj; = P?) and scalars a > 0,6 > 0, Wj; = 0,v; = 0,0; = 0,9; = 0, and njj, 
solving the optimization problem 
minf 

al + WiQi < Pi < BI— viQi 

ATP, + PjA+0Q; < —I 

Pj = Pi + ni j(Q — Q) 

91:Qi +--+ + 9nQn = 0 
foralli,j € {1,...,N}, then the switched linear system (Equation 30.1) can be exponentially stabilized (with 
decay rate xp) by the largest region function strategy (Equation 30.17). 


S.t. 
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The extension of the synthesis method for continuous-time switched linear systems to discrete-time 
counterpart is not obvious. The main difficulty is that, unlike the continuous-time case, discrete-time 
switched systems do not have the nice property that the switching occurs exactly on the switching 
surface. Instead, the switching happens in a region around the switching surface. As a result, we 
cannot simply capture the switching instants for discrete-time switched systems at the time instants 
when the state trajectories cross the switching surfaces. Therefore, in order to guarantee the nonin- 
creasing requirement at the switching instants for the discrete-time case, we need to include more 
constraints involving state transitions for the discrete-time switched systems around the switching 
surfaces. This makes the switching stabilization problem for discrete-time switched systems more 
challenging. 

Some remarks are in order. First, for both the continuous-time and discrete-time cases, the optimization 
problem above is a Bilinear Matrix Inequality (BMI) problem, due to the product of unknown scalars 
and matrices. BMI problems are non-convex, and not computationally efficient. However, practical 
algorithms for optimization problems over BMIs exist and typically involve approximations, heuristics, 
branch-and-bound, or local search. One possible way to solve the BMI problem is to grid up the unknown 
scalars, and then solve a set of LMIs for fixed values of these parameters. It is argued in [11] that the 
gridding of the unknown scalars can be made quite sparsely. 


Example 30.3: [11] 


To illustrate the synthesis procedure, consider the case of two unstable subsystems given by 


1-5 1 0 
wale v} [5 Sf 


It can be shown that there is no stable convex combination of these two matrices, which means that 
the system cannot be quadratically stabilized. However, solving the BMI in Theorem 30.18 through 
gridding up the unknown parameters results in a solution 


B= 3.7941, «w=0.2101 


and 
—0.08242 0.8648 
a =-a=[ 0.8648 aoe eure) 
_ [1.1896 1.1440 _ | 3.3325 —1.1044 Bers 
11141440 3.2447|' “2~1|-1.1044 1.1509 |° : 


Hence, the switched linear system can be exponentially stabilized by the largest region function 
strategy (Equation 30.17), and the estimate of the exponential convergence becomes ||x(t)|| < 
4.2495e—9-1318t 1 ¥5 1), 


So far, we have only derived sufficient conditions for the existence of stabilizing switching signals for 
a given collection of linear systems. A more difficult problem has been the necessity part of the switching 
stabilizability problem, and a particularly challenging part has been the problem of finding necessary and 
sufficient conditions for switching stabilizability. In [7], a necessary and sufficient condition was proposed 
for the existence of a switching control law (in static state-feedback form) for asymptotic stabilization of 
continuous-time switched linear systems. 
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30.5 Conclusion 


In this chapter, we discussed, by necessity, a brief introduction to the basic concepts and results of the 
field of stability and stabilizability of hybrid systems. For further references, we would suggest several 
survey papers on the stability of hybrid and switched systems, for example [3,6,8,9]. 
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People, engineers, and scientists encounter a variety of switched systems everyday. Gear selection in 
automatic transmissions [1], control of robots subject to constraints [2], power management in hybrid 
electric vehicles [3], load balancing in a computer cluster [4], coordination of flexible AC transmission 
systems (FACTS) devices [5], and output voltage regulation in DC/DC converters [6-10] are but a few 
examples. Minimizing energy/power usage and/or tracking errors are typical objectives in these control 
systems. These objectives are achieved by formulating an appropriate cost function to be optimized over 
some switching functions in combination with the usual continuous control input. Switched systems 


31-1 
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are a subclass of hybrid systems in which both discrete (switching) and the continuous control inputs 
are present. Additionally, there may be internal or uncontrolled switches that result when the system 
trajectory and/or continuous inputs enter certain regions of the state and input spaces, respectively. 
These switches are called autonomous. 

There are a variety of techniques for solving hybrid optimal control problems. In general, the problem 
can be subdivided into three tasks: (1) finding the optimal sequence of switching instants, (2) finding the 
optimal sequence of discrete input modes, and (3) finding the optimal value for the continuous control 
input. For general cases, completing all three tasks is difficult. For example, Giua et al. [11], Xu and 
Antsaklis [12], and Loxton et al. [7] fix the switching sequence, and then compute the continuous control 
input. Bemporad and Morari [13] suggest using mixed integer programming to find the optimal solution 
whose computational complexity is NP-hard and increases exponentially with the number of modes. 
The minimum principle [14-16] has also been applied to solve for solutions. The dynamic programming 
approach adopted in [1,17,18] has the curse of dimensionality as a drawback. In [19], the Heaviside 
function (calculus of variations) is used to re-create a continuous system from the hybrid system with 
state jumps. 

Based on the result in [20], we show, in this chapter, that for quite a general class of hybrid optimal 
control problems, the computational complexity of the problem is no greater than that of smooth optimal 
control problems. In Appendix A, we also describe how to extend the embedding methodology of [20] 
to incorporate hybrid behavior stemming from memoryless autonomous switches that results in plant 
equations with piecewise smooth vector fields. Further we point out that the approach from [20] can be 
readily extended to systems with an arbitrary number of modes with only a linear increase in complexity 
[21]. A direct collocation approach to the solution of the hybrid optimal control problems is summarized 
in Appendix B. 

The switched systems studied herein exhibit two types of switching behavior: autonomous (uncon- 
trolled) switches and controlled switches. They both result in discontinuous jumps in the vector fields 
governing the evolution of the continuous state of the system. In the case of autonomous switches, the 
vector fields of the system undergo discontinuous jumps as a result of the state and the input entering 
different regions in the combined state and input space. Such switches are uncontrolled, meaning that 
the switches cannot be affected directly through a separate switching mechanism. An example of a sys- 
tem with autonomous switches is the one subject to continuous state-dependent constraints, where the 
autonomous switches correspond to different combinations of constraints that are active in a particular 
continuous state. A practical example is a mobile robot that encounters a patch of ice in which rolling 
motion changes abruptly to sliding motion. The second type of switches involves discontinuous jumps in 
the vector fields that can be directly controlled, and thus are called controlled switches. An example of a 
system with such switches is a continuous control system whose control mechanism consists of a finite 
number of different continuous controllers, and the controller to be used is determined at a supervisory 
or decision-making level of the overall control system. Such would be the case for power management in 
a hybrid electric vehicle, or, for example, a large ship’s propulsion/electrical system. We assume that the 
set of different operating regimes of the system defined through the autonomous and controlled switches 
is finite. 

As mentioned, in hybrid electric vehicles, the energy or power management problem is naturally 
modeled as an optimal switched system in which the electric drive operates either as a propelling machine 
or as a generating machine for recharging the battery. A suboptimal model predictive control (MPC) 
approach was adopted to solve for the energy management solutions in [3,22-24]. A nonlinear MPC 
(NMPC) version provides more accurate hybrid electric vehicle (HEV) solution but at the expense of 
computational requirement [3]. To improve the computational time, a few optimization subproblems 
can be solved offline and the solutions stored as maps, thereby decreasing the computational load [25,26]. 
Real-time MPC is detailed for boost converters in [8-10]. 

The formal definition of the switched optimal control problem (SOCP) is provided in Section 31.2. 
This section also illustrates mathematically how to parameterize the family of problems as the embedded 
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optimal control problem (EOCP). This section suggests solving the SOCP via first solving the EOCP. The 
benefits of the embedding technique are also provided. 

Section 31.3 summarizes sufficient and the necessary conditions for solvability of the EOCP. It will 
be seen therein that sufficiency conditions can be met rather easily by many systems. The necessary 
conditions can be seen as a generalized Maximum Principles without explicit assumptions on fixing the 
number or sequence of switchings. This is followed by a description of the extension to three modes (and 
implicitly to n) in Section 31.4. 

The application of the necessary conditions for solving the EOCP is illustrated in Section 31.5. Therein 
a simplified vehicle with two gears is investigated [20]. Through the usage of the generalized Hamiltonian 
one can apply the EOCP to characterize the solutions, including the singular cases. 

Since applying the necessary condition requires solving for the state and the adjoint equations simulta- 
neously, the sensitivity of the solution on the initial state or adjoint state is known to be large, precluding 
the use of single shooting methods. Section 31.6 applies the direct collocation method, a more robust 
technique to solve an example of MPC of a unicycle. A summary of the direct collocation method 
is provided in Appendix B. Concluding remarks and further pointers to the literature are given in 
Section 31.7. 


31.2 The Switched and EOCPs 


This section formalizes a model and problem formulation for a two-switched system. Generalization to 
3-modes is described in Section 31.4 and the extension to n-modes is straightforward [21]. The well- 
known SOCP is developed first. Then as per Bengea and DeCarlo [20], the embedded formulation is 
presented. The relationship between the two formulations is then set forth. One can then show that the 
embedded problem is a viable approach for solving the SOCP and in contrast to other approaches has 
linearly increasing complexity with the number of modes rather than combinatorial complexity. 


31.2.1 Model of a Two-Switched System 


The state dynamics of a two-switched system is 
s(t) = frs(ey(t xs(t), us(t)), xs(to) =x ER", t= to (31.1) 


where (1) the continuously differentiable vector fields, fo, fj : R x R” x R” — R", specify the dynamics of 
each of two possible system configurations; (2) the classical control input, us(t) € Q C R”, is constrained 
to the bounded and convex set, Q2, at each time instant; and (3) a mode switching mechanism, represented 
through the subscript vs(f) € {0, 1}, identifies which of the two possible system configurations, fo or fi, 
is operational. These “controls” ensure that the system vector field can be controlled both through a 
selection of the control and switching inputs. We also assume that the initial time, fo, initial state, xs(t), 
final time, ff, and final state, xs(ty), are restricted to a boundary set B as follows: (to, xs(to), t¢, xs(tf)) € 
BT) x Box Te eBe Re, 

In addition to controlled switches, there are often autonomous switches and even switches that are 
only possible in certain regions of the state space. Appendix A describes how autonomous switches can 
be easily incorporated into the framework set forth in this chapter. 

In general, the behavior of switched systems can be quite complex and might lead to anomalies such 
as Zeno behavior or deadlock states. Furthermore, the systems of Equation 31.1 (or Equations 31.42 and 
31.44) belong to the class of systems with discontinuous right-hand sides [27]; hence, the questions of 
existence and uniqueness of solutions have to be carefully studied. Although certainly important, these 
issues are outside the scope of this chapter and we will assume that the existence and uniqueness (in the 
appropriate sense) are guaranteed. We refer the interested reader to the conference series [28,29], special 
issues [30,31], and [32,33] for further reading. 
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31.2.2 Performance Index and the SOCP 


The extent to which it is preferable (when the same trajectories can be generated in both modes) or feasible 
(when only one mode can generate desired trajectories) to modulate the continuous inputs or choose the 
switching control input value depends on the input and state constraint set and a performance index (PI). 
For measuring the degree of optimality of triplets (us(-), vs(-), xs(-)), we introduce the following cost PI: 


t, 
Is(xo, us, vs) = g(to, xo, rapt f Fy) (t, xs(t), us(t)) dt (31.2) 


to 


where the function g penalizes the endpoints and is defined on a neighborhood of B, and the integrands 
Fo and F; are real-valued continuously differentiable functions that penalize the running cost in each 
mode, respectively. Depending on the engineering applications, the functions Fy and F; represent the 
cost of operation of various subsystems that are active only during individual modes of operations. For a 
hybrid electric vehicle example, during acceleration we want to minimize fuel consumption, while during 
braking one tries to maximize the regenerative power [3,24]. 

The previously introduced dynamics, input and state constraints, and performance measure define the 
SOCP: 


min —_Js(xo, us, vs) 
us€Q,vs€{0,1} 


constrained by 


1. Dynamical state equation 31.1. 
2. Endpoint constraints (to, xs(to), tf, xs(ty)) eB. 


In the selection of the above switching system dynamics and performance measure, we make several 
transparent assumptions that limit the application of the proposed approach: 


1. The dynamical equation 31.1 models the switching control input vs(t) as an independent control 
input; therefore, the mode switching can occur independently of the state values and control 
input values. In practical examples, this may not be the case as would be for the speed-dependent 
gear-switching mechanism of a vehicle. The state-dependent switching can still be studied via the 
formulated SOCP by penalizing the switching (a formulation known as a “soft” constraint). 

2. Mode-switching is assumed to occur instantaneously. In engineering application, switching among 
the vector fields is implemented by generating new actuator values that interconnect and/or dis- 
connect various subsystems. In power electronics, such switching takes place over a time interval 
that is negligible in comparison with other component dynamics. When this switching action 
absorbs large amounts of energy or affects the dynamics of the physical system beyond the con- 
nection/disconnection process, the switching dynamics may need to be modeled. 


31.2.3 The Embedded Optimal Control Problem 


To display the interaction between the discrete and continuous control inputs Equation 31.1 can be 
rewritten as a convex combination: 


Xp(t) = [1 — ve(t)] - fo(t, xe(t), uzo(t)) + ve(t) - filt, xe(t), ues (t)) (31.3) 


which reduces to Equation 31.1 under the conditions: vg(t) = vs(t) and ugo(t) = ug (t) = us(t). However, 
although Equation 31.3 is equivalent to Equation 31.1 under the indicated conditions, this new equation 
invites exploration of additional trajectories generated with an enlarged domain of the switching control 
denoted here by vg € [0, 1] and independent continuous-time controls ugo(t), ugi(t) € Q. 

For illustrating the possible trajectories generated with vg € [0, 1], we sketch in Figure 31.1 vector fields 
and trajectories for a hypothetical switching system and the corresponding switching controls vg. With 
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FIGURE 31.1 Trajectories generated with various control inputs vg for a hypothetical switch system: vg = O(solid 
upward arrow), vg = 1 (solid rightward arrow); some vz € (0, 1) (solid dark arrow). 


this enlarged control input set the original switching control becomes a special case; it would appear that 
the generated trajectories are no longer feasible for the original switched system. 

This expansion of the control input domains follows relaxation techniques from optimal control 
employed when certain sets that guarantee solution existence need to be rendered convex [34]. We call 
this relaxation of the switching system an “embedding.” The PI of Equation 31.2 has a similar embedding 
given by 


t 
Jos aoe Ye) = Bore (a) tp see) + f {[1 — ve(t)] - Fo(t, xz(t), uzo(t)) 
to 


+ ve(t)- Fy(t, xe(t), wei (t))} dt (31.4) 
which leads to the EOCP: 
min Je (Xo; UE0> UE VE) 
UED UE EQ, VE E[D,1] 
subject to 


1. Dynamical state equation 31.3. 
2. Endpoint constraints (to, xz(to), tf, xE(ty)) EB. 


The EOCP becomes now a classical optimization problem with continuous control inputs. Although 
application of established techniques is now possible for solving this problem, the EOCP’s solutions 
(xf, Ufo» Up] Ve) may have vz € (0, 1) for almost all time instants t € [to, tf]. One wonders how this might 
help solve the original SOCP where the switching control v¢ € {0, 1}. Ifone selects a signal vs that switches 
between 0 and 1 with an appropriate duty cycle per (small) unit time, then a trajectory of Equation 31.1, 
so generated, ought to approximate an embedded trajectory x, as is illustrated in Figure 31.2. The idea 
of approximating trajectories generated with vg by appropriately selecting an on-off signal vs € {0, 1} is 
made more rigorous in Theorem 31.1. 


Theorem 31.1: 


Let ugo, Ug, € Q and vg € [0,1] be a control triplet for the embedded system Equation 31.3 and xg the 
generated trajectory on the interval [to, tf]. Let both the switched and embedded systems have the same 
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FIGURE 31.2 Embedded system trajectory xg(black), approximating switching system trajectory xs, and corre- 
sponding switching signal. 


initial condition, xs(to) = xg(to). Then for any desired trajectory-approximation error ¢ > 0, there are 
control inputs vs¢(t) € {0, 1} and us.(t) € Q defined on [to, tf] such that the generating switching trajectory 
has the property l|xs,(t) — xg(t) | <e for allt € [to, tf]. 


The proof of the theorem and details on the approximation can be found in [20]. By establishing 
that the set of embedded system trajectories is not significantly larger than the set of the trajectory set of 
Equation 31.1, the theorem validates the approach of generating optimal solution for the SOCP by solving 
the EOCP, a classical optimization problem. Indeed, the set of trajectories of Equation 31.1 is dense in 
the set of trajectories of the embedded system. After application of standard techniques for generating 
a solution (xf, Upp, Up), Ve) for the EOCP, at least a suboptimal solution of the SOCP (xs, us, vs) is 
guaranteed to exist by Theorem 31.1. How best to achieve the approximation remains an area of open 
research. 

Based on Theorem 31.1 and on the properties of the EOCP, it can be shown that the relationships 
of Table 31.1 hold (see Propositions 3 and 5 in [20]). An alternate proof of Theorem 31.1 based on the 
Lyapunov (integral) theorem is given in [35]. 

The embedding approach enables the study of the cases when the SOCP does and does not have 
solutions. The presence of singular solutions, with vj € (0, 1), can reveal situations when the SOCP does 
not have solutions. The only case, 2a in Table 31.1, when the EOCP does not solve the SOCP is the 
case when the end-state of the embedded system trajectory x; (ty) is on the boundary of the closed set 
Br. And this happens because the end-states of approximating trajectories xs.¢(t7) are not guaranteed to 
meet this final constraint. This case has not been completely studied, and the current analysis does not 


TABLE 31.1 Relationship between SOCP’s and EOCP’s Solutions 


EOCP Solutions SOCP Solutions Remarks 
1, Bang-bang solutions The same as ECOP SOCP is solved 
with vz, € {0, 1} bang-bang solution with 
ve = vp and x¢ = xp 
2a. Singular solutions only May have solutions EOCP does not solve SOCP 
VE € (0,1) 
2b. Does not have solutions Suboptimal solutions can be 


constructed via Theorem 31.1 
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exclude its existence. From an engineering perspective, however, this case has little relevance. By using 
engineering approximations for a particular application, the terminal constraint set can in most cases be 
slightly enlarged making Case 2a unlikely. Hence, approximate feasible solutions of the SOCP can always 
be constructed, motivating a continued focus on the embedding approach. 


31.3 Sufficient and Necessary Conditions for Solvability 
of the EOCP 


This section presents conditions that guarantee the existence of EOCP solutions and properties of these 
solutions. Combined with the EOCP and SOCP relationships of Table 31.1, the existence conditions 
for the EOCP solutions enable the study of sufficient conditions for the SOCP. In deriving existence 
conditions for the EOCP solutions, we employ the generic theorem of Berkovitz [34, Theorem 51, p. 61]. 
With appropriate notational adaptations to the EOCP formulation, this theorem states that a solution 
exists for a generic optimal control problem, if the following conditions are met: 


The set of admissible pairs states-control inputs, (xg, Ugo, UE1, VE), is not empty. 
There is a compact set that includes all the points (f, xg(t)) for all t € [to, tf]. 
The terminal constraint set B is compact. 

The input constraint set 2 x Q x [0, 1] is compact. 

The set 


GY eG): oN te 


Qt - {(y.y) :y? > (1 = Ww) Fo(t, XE; UFO) + WF) (t, XE, UuE1)s 
y= (1—p)fo(t, xz, uzo) + wilt, xe, vei), € [0, 1], uz0, we € Q)} (31.5) 


is convex. 


Among all the above conditions, meeting the convexity requirement of the set Qf is the most chal- 
lenging. From [34], this condition is met when a system is affine and the penalty cost is convex in the 
continuous control input. The particular form of the EOCP, where both the vector field and the penalty 
costs are affine in the additional switching control input, makes possible the extension of the mentioned 
results to the EOCP. 


Proposition 31.1: 


The set OF; defined in Equation 31.5, is convex for the following class of EOCP vector fields: 

(S1) The vector fields for the two modes of operation are linear in their control inputs: fo(t, Xz, Uz0) = 
Ao(t, xg) + Bo(t, xg): uso, filt, xg, Ue) = A(t, xe) + Bi(t,xg)- ug, with Ao, Bo,A1,B, continuously 
differentiable functions. 

(S2) The cost functional integrands are convex in their continuous control inputs: for every (t, xg); the 
functions Fo(t, xg, Ugo) and F,(t, xg, ug) are convex in the inputs ugg and ug}, respectively. 


Proposition 31.1 and the previously presented relationships of Table 31.1 between the SOCP’s and 
EOCP’s solutions are the main implements one can use for analyzing the existence of SOCP’s solutions. 
Before understanding what particular case of Table 31.1 is applicable to a specific problem, the EOCP 
solutions must be generated. Assuming that the EOCP has a solution, a characterization of these solutions 
can be made by using classical results of optimal control theory. In the following, we use these results and 
the particular form of EOCP vector fields and cost functional to derive conditions corresponding to the 
cases presented in Table 31.1. 
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The Hamiltonian associated with the EOCP is a function Hg : R x R” x R™ x R™ x [0,1] x Rx R" > 
R defined as 


He(t, xg; Uno, Ue, Ves AB XE) = AP [1 — vE)Fo(t, xe, Ueo) + VEF i (t, Xe, Ue1)] 


+£[(.—ve)folt, xe, ueo) + vefi (ts xe, ue1) | (31.6) 


To emphasize the affine dependency on the generalized switching input vg, the Hamiltonian is 
factored as 


He(t, Xp; UEo UE, VE, Mf ME) = Eo(ts XE, UE» UEIs Mes hE) + VE + Ey (t, Xp, UE, UEI> Mos KE) (31.7) 


with the obvious definitions of Ey and E;. Assuming that an optimal solution of the EOCP exists, denoted 
hereafter as (t, x}, Ui, Ue), Vz), a characterization of its properties is possible via application of Maximum 
Principle [34, Theorem 31, p. 185, and Corollary 3.1, p. 186]. These conditions guarantee the existence of 
a constant ie and an absolutely continuous function 4;,(f) on [to, tf] such that for almost all t € [fo, tf] 
the following hold (the argument f is not included for simplicity): 


.x OE 
YS 
* Oke Jspautout wpa ag) 
(NC1) 
sx _ _ OHe 
tee OXE KR gpk yk 9 OF 4K 
(XE MEQ MEDVE NE AE) 


where (OHg/0,g) and (OH_/Oxg) denote appropriate partial derivatives; 
He(t, xf3 Upgs Upy> Vis Me's ME) = Het, xf, UEos UE VEs NBs NB) (NC2) 


for all uzo, Ug, € Q and vg € [0, 1]. Using simple manipulations detailed in [20], condition NC2 can be 
shown to be equivalent to 


* Ok * * 4 Ox 4 * 
Hg(t, XE» UEQ UEP VE> ME » dz) 


= max{ max {Hg(t, xp, Ugo, UE; 0, es Ne)}, max {He(t, xf, Ugo, Ue 1, rae a) (31.8) 
UE0UEL UEOUEL 

where the two inner Hamiltonian expressions are directly associated with the two modes of the switching 

system. We observe this is the first step in the derivation where the SOCP’s solutions emerge as solution 

of the EOCP. However, Equation 31.8 alone does not yet guarantee that EOCP solutions are of the bang- 

bang type, that is, vz € {0,1}. In analyzing the cases when the EOCP has bang-bang-type solutions, the 

expression 


0 
E, (t, XE, UE0> UE1> Nps KE) 


(31.9) 
= df [Fi(t, xg, ue1) — Fo(ts xe, uzo)] + Xf [fil xe, unr) —fo(t xe» Ueo) | 


of Equation 31.7 plays a critical role, specifically when it becomes zero. This role motivates the introduction 
of the following set of time instants: 


TS {t € [to, t¢] : Ex(t, xp, UEo» UBL VED nore) = o} (31.10) 


As an intuitive explanation, suppose the expression E of Equation 31.9 is nonzero almost everywhere 
along the optimal solution then its sign would indicate which of the two maxima of Equation 31.8 is 
attained. This in turn specifies the optimal value of the control input vz, which in this case would be 
restricted to the set {0,1}. These statements are formalized in the following theorem: 
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Theorem 31.2: 


For almost allt € (to, t7] — T the following hold: 


1. (Mode 0) If maxXupo,up, {HE(t Xp VEO UE1, 0; nes MB)} > MaXugo,ug, (HE(t Xp» UE, VEL 1, Me AB} 
then v;,(t) = 0 and the equations of NC1 become 


XE = fo (t.$, whe) 


: OF a (31.11) 
sal ALE 
Om Meith) OX J txf,ufy) 
The optimal control input uj, has an indeterminate value and 
Ujy = arg min {Hp(t, x%, ugo, ug1, 0, O*, X%)} (31.12) 
UuEDEQ 


(here HE does not depend on uf). 
2. (Mode 1) Tf MaXupgo,u¢1 {HE(t, Xp uE0> UuFl> 0, A, ME} < MaXyp9,uz1 {HeE(t, XB UE0> UEl> 1, ens MB} 
then v;,(t) = 1 and the equations of NC1 become 


ap =fi (6p “h) 


<x —_,0[ OF _,r fh (31.13) 
= sd Bee (t.x%,u%, ) E | Ox (t,x%,u%s, ) 
EEL 2 FOE] 


The optimal control input uj) has an indeterminate value and 


up) = arg min {He(t xp, ueo, vers 1, RE) (31.14) 
uELEe 


(here Hg does not depend on ufo). 

3. (Nonsingularity) If MaXup,up, {HE(ts Xp UO. UE 0, pe NME)} = MaXyp9,up, (HE(t, Xp UEO, WEL 1, Ha 
XE)}, then either vj € {0,1} or the corresponding . Equations 31.11 or 31.13 hold, but additional 
constraints need to be used to determine exactly which of the two modes are optimal at time t. 


Theorem 31.2 characterizes all the situations of Case 1 of Table 31.1. The singular solutions, summarized 
in Case 2 of Table 31.1, may be generated when the expression E; of Equation 31.9 equals zero for some 
time interval of nonzero measure; in this case 

max {He(t, xg, Uzo, UE1,0, Xp Kp)} = max (He(t, xp, ueo, wer 1, Mp" AE) (31.15) 
UEOUEL UEO>UEL 

As such, the optimal solution xf is generated with some vz, ¢ {0, 1} on the same nonzero measure time 

interval. In summary, all these cases occur when the set T of Equation 31.10 is positive time-invariant, 


on some time interval, for the dynamical system of NC1, and Equations 31.12, 31.14, and 31.15 hold 
simultaneously for some vz ¢ {0, 1}. 


31.4 Optimal Control of a Switched System with 
Three Modes of Operation 


The results of Section 31.3 can be extended to a switched system with multiple modes of operation, 
and we illustrate here the extension to three modes only for simplifying the notational complexity. Let 
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the vector fields of these modes be denoted as foo(t, xs, us), foi (ts Xs, us), and fi (t, xs, us). The subscripts 
are selected to correspond to different combinations of two discrete signals (vso, vsi) € {0, 1} x {0, 1} as 
described below by the dynamics of the switched system (where we drop the time dependency to simplify 
the exposition): 


xs = (1 — vso) « {(1 — vs1) - foo(xs, us) + vs1 « for(xs, us)} + vs0 -filxs, us) (31.16) 


One observes that, for example, the mode-switching input combination (vso = 0, v5] = 1) selects the 
dynamics xs = foi (xs, us), and therefore three modes are possible: 00, 01, and 1. Other vector field com- 
binations can be selected to describe the same switched system dynamics: 


Xs = Vso « foo(xs, us) + vs1 - for(xs, us) + (1 — vso — vs1) - filxs, us) (31.17) 


Although the results are the same for the two approaches, the one described by Equation 31.16 has two 
advantages: 


1. A parsimonious selection of switching inputs. For n modes, Equation 31.16 uses at most [log, n| 
inputs (the smallest integer larger than log, n), whereas Equation 31.17 uses (n— 1) inputs. 
Although in our case, for n = 3, [log, n| = n-— 1, the number of discrete inputs becomes larger 
for Equation 31.17 as n increases. 

2. The results of Sections 31.3 and 31.4 can be directly applied in a few nested steps. 


We continue to use Equation 31.16, and refer the reader to [21] for development of similar results 
derived based on Equation 31.17. The embedding of switched system of Equation 31.16 follows the 
same approach as presented in Section 31.4 with the switching control inputs redefined as (vz, ve1) € 
[0, 1] x [0, 1]: 


ig = (1— veo) - {(1— vex) - foo(xe, uzoo) + ver - for (xe, Ueo1)} + veo «fi (XE; WEI) (31.18) 


Intuitively, and this is based on the construction theorem presented in [20], Theorem 31.1 holds for 
systems (Equations 31.16 and 31.17). This motivates the study of the EOCP and we focus on the necessary 
conditions for the embedded system solution (x2, Uiq9> Ug,» UEy> Vg» VE1) to be a solution of the SOCP, 
with (Vp; VE) € {0, 1} x {0, 1} and x¢ = xf. 

With the optimization cost integrands, similarly to Equation 31.4, denoted by Foo (xz, UE00)» 
Fo, (xz, UE01)> and F; (xz, ug1)> the Hamiltonian of the embedded system (Equation 31.18) can be written 
as 


Hg (xz, UE00» UEOL, UELs VEO> VEL> Mf» ME) = XE {(1— veo) - [(1 — ver) « Foo(xe, uzoo) + vet - Foi (xz, uz01)| 
+vz9 * Fi (xz, ue1)} + Xf > {(1 — veo) - [(1 — ver) - foo(xe; uz00) 
+ve1- for (xe, Uz01) | + veo «fixe, uei)} (31.19) 


Similarly to Equation 31.7, the three-mode switched system Hamiltonian of Equation 31.19 can be 
rewritten more compactly as 


He = veo: Ve. Ey + vei Ex + ve0- E3 + E4 (31.20) 
where the arguments (xz, UE00> UEO1> UE1> VEO: VEL> ie he) are dropped for Hg and the new expressions E), 


E>, E3, and E4. These expressions are very similar to those of Equation 31.7: they are weighted summations 
of differences between vector fields and penalties corresponding to pairwise switching modes. These 
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expressions are 


E, = df - [Foo — Foi] + Xf - [foo — for] (31.21) 
Ey = [Fi — Fool +22 - [fi — foo] (31.22) 
E3 = f° [Fo — Foo] + Xz - [fo1 — foo] (31.23) 
Eq =f  Foo + dg - foo (31.24) 


where again the arguments are dropped. 

The compact form of the switched system dynamics that resulted in the Hamiltonian of Equation 31.20 
can now be used to generate the final result of this section. The following necessary condition for the 
optimal solution (x%, Uxqq> Ub.» Uey> Veo» Vez» AG's Kf) is derived by applying in two steps the results of 
Lemma 10 of [20] in two steps (again, the arguments are not included). 

HE (Optimal Sol) 1 max HE (xz, UEQO» UEOL» UELs VEO> VEL» dos Ne) 


(vg0.VE1)€[0,1] x[0,1] 
(ugo0,UE01 UE1)EQXQX Q 


= max {veo - Ve. *E, + vg - Eo + ve0 - E3 + Ey} 
(vg0.VE1)€[0,1] x [0,1] 
(ug00,UE01 UE )ENXQXQ 


max (ve. E2 + Ey}, 
veo=0,vE1 €[0,1] 
(ug00.UE01 UEL)EQX AX Q 


= max 
max {ve1- Ey + vei: Eo + E3 + Es} 
veo=1,ve1€[0,1] 
(uz00,UE01 WEI )EQXQXQ 
nee Lt on {Ea}, max {Ex + Ea} ¢, 
vg0=0,VE1=0 vE0=0,VE1=1 
= max (ug00,UE01 MEL )ERXQXQ (ugo0.MEOLUEL)ERX QAXQ 
max max {E3 + E4}, max {E, + Ey +E3+ Es} 
veo=1,vE1=0 vpo=lvp1=1 
(uz00HE01 HEI EL XQXQ (ug00,UB01 UE )ENXAXQ 
max Ej}, max (Ey + Ea), 
(ugoo ME MEL )ERX QX Q (ugoo MEI UE )ERXAXQ 
Saw (we used E, = —E3) 
max {E3 + Ey} 
(ug00,UE01 UE )EQXQXQ 


This is an equivalent way of saying that the embedded-system Hamiltonian calculated at the 
optimal solution is equal to the maximum of the Hamiltonians associated with all modes, similarly to 
Equation 31.8. 


* O* 4 > 
tax He (xg, Uz003 UE01» WEL. VEO = 0, VE1 = 0, Xe NE) > 
(uz00.ME01 UE )JEQXAXQ 
A mM: m H O* 4 > 
: = max E Xb5 UEOOs UEOL> UEI>» VEO = 0, VE] = 1,0°, x35), 
‘ (Opti ss Sol) AX) (uzo0.ueo UE JELXQXQ ( E E z) 


* — Ox 4 > 
max Hg (xf, UE00> UE01» UE VEO = 1, vg) = 0, Xp » hz) 
(ug00.ME01 HEL )JEQXQXQ 
(31.25) 


Therefore, a result similar to Theorem 31.2 can be derived providing necessary conditions for optimality 
for the embedded system of Equation 31.18. 


31.5 Two-Gear Car Example with Application 
of Necessary Conditions 


This section uses the embedded methodology developed in Sections 31.2 and 31.3 to analyze a crude two- 
dimensional model of a car with two gears [1, Example 1, p. 3975] having speed-dependent efficiencies 
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Gear efficiencies 


Speed (x») 


FIGURE 31.3 Gear efficiencies as a function of normalized speed. 


g0(¢) and gi(c) as plotted in Figure 31.3. Let x; denote the car’s position and x its velocity with respect 
to some coordinate system. 
The embedded system has the form (x = (1 — v)fo(x, uo) + vfi(x, 1): 


x(t) ay ee x(t) x9(t) 
| = (1-v(t)) E (x2(t)) - eal + v(t) i (x2(t)) al (31.26) 


The inputs uo(t), w(t) € Q =[—1,1] represent control of the brake or throttle. The terminal state 
constraints are x(0)=[—5,0]! and ae) =[0,0]’ in which case the terminal constraint set is 
B= { (to. x0» tp xf) -_ (0, [—5,0]", tf, [0, 0] ) ty € R}. Additionally, we require that the vehicle begins 
in Mode 0 and returns to Mode 0 at ty, but for the moment we do not impose these constraints; it turns 
out that they follow directly from the solution of the “mode unconstrained” scenario. Finally, we define 
the embedded PI to be 


tf t 
TE(x, Vv, Ug, Uy) = fe [(1 — v(t))Fo(t, x, u) + v(f)Fi(t, x, u)] dt = / : dt = tf 
0 0 


where we have set Fy = F, = 1; this is a minimum time problem. One observes that the sufficiency 
conditions for the existence of an optimal solution are satisfied. We denote the optimal solution by 
(x*(t), up (t), uf (t), v*(t)). 

The Hamiltonian associated with system Equation 31.26 is with }—=[A1,d2]! (suppressing 
t-dependencies) 


He (t,x, Uo, U1, Vs 2°, ®) = V* Ao [gi (x2)u1 — golx2)uo] + [K° + Aix + d2go(x2)uU0] 


‘ ‘ (31.27) 
4 y-Ey(t,x, uo, ur, d°, d) + Eo(t, x, uo, ui, %°, d) 


We like to argue that the Hamiltonian of Equation 31.27 implies the existence of bang-bang solutions as 
wellas other information about the optimal solution. This requires that Ej (¢, x*(t), up (t), ut (t), n°, X(t) #0 
almost everywhere on [0, ty], which is shown in Proposition 4.2.1 [36], that is, there are only bang-bang 
solutions to the hybrid optimal control problem. Let us now determine the values for the optimal solution. 
From the material of Section 31.3, there exists a constant ” < 0 and an absolutely continuous function 
XC) : [0, tp]> R? such that (2, dt) # 0 on [0, tf], and for almost all t € [0, tf], the state equations 


BG = (8) 31.28 
Be eetipeii 1 voRues te 
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and costate equations (suppressing t-dependencies) 


. 0 

"| 

Se ee dgo dgy (31.29) 
B Ay — A2(1 — v*) UG | a — hov*ut E r 


hold. 
We can draw two conclusions: (1) from Equation 31.29, \4(t) = 41, a constant that can be shown to 
be greater than zero; and from the transversality condition, 


Hg (ty, x* (tp), up (ty), ut (ty), v* (ty), 0°, M(t) = 0 (31.30) 


Thus, Equations 31.27 and 31.30, with x3 (ty) = 0, imply that 


r+ alte) [v* (gi (Ou (tp) + (1 — v* (tp) g0(0)uG (ty) ] = 0 (31.31) 


With Equation 31.31 and a “proof by contradiction,” one can also show that \2(-) 4 0 (not identically 
zero) on any nonzero subinterval of [to, ty]. Hence, using an equivalent form of Equation 31.8, we conclude 
that 


max Hg (t, x*(t), uo, “1,0, 0°, X(¢)) — max He (t,x*(t), uo, ui, 1, °, X(£)) 
ugpEQ u,EQ 


= pe + ad (t) + ; max [ra(t)go (x3 (¢)) wl = tS + 1x5 (t) + ; max [ra(t)gi (x3 (¢)) ul} 


= [go (23) — a1 (2(0)] max, [oul (31.32) 
From Equation 31.32, we observe that the optimal mode of operation is the mode with the largest gear 
ratio, and, excepting the time instants when (possibly) \2(t) = 0, the optimal throttle/braking control is 
either +1 or —1, depending on the sign of i2(f). This brings us to the following optimal operation: 


1. Similar to the discussion in [1], the optimal mode v(t) is given by the mode with the largest 
efficiency at time instant tf. 

2. For Mode 0 (Ej(-) > 0). If go (x3 (t)) > gZ1 (x3(t)), that is, x}(t) <0.5 for some ¢ € [0, tf], then 
v(t) = 0 and 

sgn (X2(t)) ; Ao(t) £0 


31.33 
indeterminate in[—1,1], 2(t)=0 : 


u(t) = | 

3. For Mode 1 (Ej(-) < 0). If go (x3 (t)) < 21 (x¥(t)), that is, x}(t) > 0.5 for some ¢ € [0, ty], then 
v(t) = 1 and 

sgn (d2(t)) h(t) 40, 


31.34 
indeterminate in[—1,1] 2(t)=0 


u(t) = | 
4, From the endpoint constraints, x3(0) = x3 (tf) =0, it follows that v*(0)= v* (tf) =0 since 
go (x3 (t)) > gi (x4(2) for t = 0, tf; thus optimality enforces the physically meaningful mode con- 
straint at the endpoints. Finally, we mention again that the constant i > 0, and that as per [36] 
there exists 0 < ft) < tf such that 2(t) > 0 for all t € [0, t;) and 2(t) < 0 for all t € (4, tf]. We 
make the following further conclusions: 
l. Ift <t, and x}(t) < 0.5, then v*(t) = 0 and up (t) = 1. 
2. Ift < t and x(t) > 0.5, then v*(t) = 1 and uj (t) = 1. 
3. Ift > t and x3(t) < 0.5, then v*(t) = 0 and uj(t) = —1. 
4, Ift >t; and x}(t) > 0.5, then v*(t) = 1 and uF (t) = —1. 
It turns out that this solution is also optimal for a penalty on switching. See Figure 31.4 for 
simulation results. A variation of this example for control through a communication network with 
fixed delay is considered in [37]. 
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The optimal trajectory in the state space 
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FIGURE 31.4 Optimal x} (t), x} (¢), u*(t), and v*(t). 


31.6 Unicycle Example with Direct Collocation 


To demonstrate the full power of the embedding approach, we show how to numerically compute optimal 
trajectories for a system exhibiting both controlled and autonomous switches. Further details on how the 
embedding technique can be applied to such systems can be found in Appendix B and [21]. The example 
considers a unicycle driving on a horizontal plane (Figure 31.5). The wheel of the unicycle can either roll 
or slide, resulting in autonomous switches. In addition, we assume that the unicycle has a regenerative 
brake that can be turned on or off. These switches are controlled. We assume that the unicycle contains a 
separate motor and a generator, both connected to a battery pack. This implies that the system can brake 
either by applying a negative torque on the wheels or by using the regenerative braking. 

Referring to Figure 31.5, the forward velocity of the wheel is controlled by the torque u; applied to the 
wheel’s axle, while its heading is controlled by the torque u2. Our objective is to drive the unicycle to 
the origin within an allotted time while minimizing weighted power usage. We use MPC [38] to compute 
the control inputs for the system. 
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FIGURE 31.5 A top view and a side view of the unicycle. 
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31.6.1 Unicycle Model 


The generalized coordinates for the unicycle are its center of mass position x and y, body orientation 6 
(heading) relative to the x-axis, and the angular position of the wheel ¢. Since we are not interested in ~ 
itself, the state variables for the system are z! =[x, V9, Ves Vy, 6, ol" € R’, where [v,; Vy] is the velocity of 
the center of mass of the unicycle, expressed in the body frame, 9 is the turning velocity of the unicycle, 
and ¢ is the angular velocity of the wheel as it spins on its axle. The equations of motion for the unicycle 
take the form 


X = Vy cos(8) — vy sin(9) (31.35a) 
y = vx sin(®) + vy cos(9) (31.35b) 
6=a (31.35c) 
F. ; 
rAd a by, (31.35d) 
m 
FE. ‘ 
ees Ae bv, (31.35e) 
m 
ers | 
6=—w (31.35f) 
0) 
. 6UEQr 1 
=—+4+-— 31.35 
g I + a uy ( g) 


where (1) m is the mass of the unicycle, (2) r is the radius of the wheel, (3) I; is the moment of inertia of 
the wheel around its axis, (4) Iz is the moment of inertia of the unicycle about the vertical axis through 
the center of mass, and (5) F, and F, are the forces between the ground and the wheel in the forward and 
lateral directions, respectively. 

The autonomous switched behavior of the unicycle occurs because F,,(z) and F,(z) depend on whether 
the unicycle is rolling (F, and F, are ground reaction forces that oppose slipping) or sliding (where F, and 
F, are frictional forces). We note that when rolling, the relative velocity, v, = [vix, Vey]? = [vx + gr, vyl" 
between the ground and the wheel’s point of contact is zero, that is, 


[Vixs Vry] = [vx + Or, vy] = [0, 0] (31.36) 


Thus from Equations 31.35a and 31.36, with j1g the coefficient of dynamic friction and g the gravita- 
tional constant i 
—mr Fess ia] Rolling 
_ bamg 
IIvrl 
The autonomous switch from rolling to sliding occurs when the magnitude of the constraint force F = 
[Fx oA exceeds the maximum possible magnitude of the static friction, j1;mg (\1s being the coefficient 
of static friction), that is, ||F|| > wsmg => “rolling — sliding”. On the other hand, the switch from sliding 
to rolling occurs when (1) v, = [v;x, Vey? = 0 and (2) the maximum magnitude of the frictional force 
exceeds that of the constraint force F, that is, ||v,|| = 0 and ||Fl| < Fymax = Wsmg => “sliding > rolling”. 
In contrast to the autonomous switches, described above, the regenerative brake can be switched off 
(Mode 0) or switched on (Mode 1) arbitrarily. For Mode 0, 


[F.(z), Fy(z)] = (31.37) 


[vx + or vy | Sliding 


u, = u°, € [—20, 20] (31.38a) 


denotes an actuating torque that can be either propelling or braking. Mode 1 denotes the use of regener- 
ative braking alone in which case 


Wy = uy = ies: bdeee (31.38b) 
—20sgn(p), |p| > 2 
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where K, = 10 is a fixed regenerative braking coefficient; note that the magnitude of u; saturates at 20 
Nm. Note that both control modes are possible in either sliding or rolling, an example of decoupled 
switches. The resulting torque control has the form 


u(t) = (1— v(t) uy(t) + v(t)uy (t) (31.39) 


where v(t) € {0, 1} denotes the original problem whereas v(t) € [0, 1] the embedded problem which is 
solved in this investigation. Because of the autonomous switches determined by Equation 31.37, the 
right-hand side of Equation 31.35 is piecewise continuous, provided the system does not chatter about 
the boundary implicitly defined in Equation 31.37; if there is such chattering then we must interpret the 
solution of the equations in the sense of Filippov [27]. 


31.6.2 Control Objective, PI, and MPC 


The objective of the control design is to drive the unicycle from a given starting initial state, z/ = 
[x(0), y(0), vx (0), vy(0), (0), «2(0), 6(0)]7 = [0,4,0,0,1,0,1]! back to the origin while minimizing the 
energy usage. In addition, we like to limit the undesirable sliding motion of the wheel as it implies a loss 
of controllability. As such, the PI takes the form 


T 
T= co llz(T) II" +f [a (1—v) (m1)? +e (u2)? +03 Il? ] dt (31.40) 


where the constant weights c; > 0. The term (1) co (z"(T)z(T)) drives the final position of the unicycle 
toward the origin; (2) cy (1—«) (uy)? penalizes the actuating power usage; (3) c2 (u2)” penalizes the 
heading power usage; and (4) c3 (II Vr I) limits the sliding motion. The terminal constraints are enforced 
through the cost functional (as soft constraints) rather than imposed as hard constraints because the 
system is stabilizable but not controllable in the sliding regime. Therefore, using hard constraints could 
make the optimal control problem infeasible. Note also that there is no penalty for regenerative braking. 

The control objective is to minimize the PI of Equation 31.40 subject to the embedded state dynamics 
given by Equations 31.35 and the initial state zo. However, when applying the computed controls to 
the actual model that differs from the nominal model due to the presence of disturbances or modeling 
uncertainties, the state trajectory might deviate from the desired trajectory, and fail to reach the desired 
final state within the allotted time interval. To cope with such disturbances and uncertainties, an MPC- 
type controller that is well known for its robustness is utilized. The MPC approach can be summarized as 
follows: 


1. Given 2, partition the time interval T into N equal subintervals of length h = T'/N, for the 
purpose of computing a (backward) piecewise constant control sequence {i, as in}, where 
aj = [uy (ih) up (ih) e and the state values {z),..., Zn}. 

2. Fork=1,...,N, solve the embedded problem of the unicycle over the receding horizon [k, N] by 
minimizing the PI given by Equation 31.40 subject to the nominal model with the initial state z,_1 
and obtain the (look ahead) control sequence {itk, ce iin}. 

3. Apply the control input a; for the time interval t,_| < t < t, to the real model. The value of the 
state of the real model at the end of the interval becomes, the initial condition for the next iteration. 

4. Repeat steps 2 and 3 untilk = N. 


31.6.3 Simulation Results and Discussion 


A variation of the direct collocation method [39] is used to numerically solve the EOCP at each step of 
the MPC algorithm. The number of points for the discretization was N = 20. Nominal trajectories are 
for the model without any disturbances. Comparisons are made to the MPC-controlled process with a 
frictional disturbance. In all cases the embedded problem has a bang-bang solution, meaning that it is 
also a solution to the original hybrid optimal control problem. 
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Mode 
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---- Rolling/sliding actual 


---» Brake on/off nominal 


sete Rolling/sliding nominal 


Off 
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Normalized time t/T (T = 9.5 s) 


FIGURE 31.6 Modes of operation. 


31.6.3.1 Unicycle Trajectories for the PI of Equation 31.40 


The plots show trajectories of the unicycle that minimize the minimum energy PI of Equation 31.40, and 
its behavior under MPC control. The computed cost of the nominal optimal trajectory is 972.2 versus 
1036.3 for the disturbed MPC-controlled trajectory. The nominal response is for the model without 
disturbances, whereas the MPC control is for the frictional disturbance where in an annulus 0.9 < r < 1.4 
around the origin, the static coefficient of friction j1, drops from 0.7 to 0.002, and the dynamic coefficient 
of friction 1g drops from 0.6 to 0.001. Moreover, the unicycle’s nominal parameters Mpominal = 1 and 
Tnominal = 4 during the simulation were perturbed so that mactyal = 1.05 and ractyal = 3.9. After starting 
in the sliding mode, the unicycle is driven to the rolling mode after about 0.2 s. At around 5s the unicycle 
encounters the frictional disturbances and switches from rolling to sliding as indicated in Figure 31.6. 
However, after leaving the slippery area at about 6s, in conjunction with the corrective action of the 
MPC controller, the unicycle starts to roll again. Figures 31.7 and 31.8 show the unicycle’s trajectories 
and the evolutions of two states V, and V,. It clearly shows that the unicycle can still reach the origin 
in the required time despite disturbances and model errors. Figure 31.9 displays the control inputs that 
again adapt in accordance with state and model changes. The results thus confirm that the influence 
of disturbances on system performance is small and the MPC scheme achieves good performance and 
robustness. Recall that the electric motor can apply both a propelling torque and a braking torque u! in 
Mode 0, while a regenerative braking torque uj is applied in Mode 1. One observes that during the final 
4s of the simulation, the switches of the braking torque in Mode 0 and the regenerative braking torque in 
Mode 1 are coordinated to reduce cost and reach the origin on time. 


31.7 Concluding Remarks 


This study has developed an approach to the solution of optimal control of switching systems that converts 
the nonconvex SOCP into a convex EOCP, which allows for direct solution to the SOCP or for arbitrarily 
close approximations to the SOCP except in rare circumstances. This chapter illustrated the use of the 
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x-y trajectories 


— Actual 
---- Nominal 
fenchos Slippery area 


FIGURE 31.7 Wheel’s position on the x-y plane. 


approach in two examples while directing the reader to the references for additional examples and for 
further developments. What was not discussed in this chapter is the development of discrete-time optimal 
control problems and the related ongoing work. To this end we offer the following recent work. 

A general framework based on the method of approximate dynamic programming is developed in [40] 
for the controller synthesis of discrete-time switched linear systems (SLS) as well as discrete-time nonlin- 
ear systems, particularly for their optimal control and stabilization. As pointed out earlier in this chapter, 
direct solution of switched system problems is challenged by their combinatorial nature, the size of 


— v,, actual 


v= v, actual 
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sees Vy nominal 


-1.5 T 1 1 T 1 1 1 1 1 
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Normalized time t/T (T = 9.5 s) 


FIGURE 31.8 Forward and lateral velocities. 
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Control torques 
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FIGURE 31.9 Control inputs. 


control policy space, which consists of traditional controls and discrete mode sequences that become pro- 
hibitively large as the time horizon increases. Using an iterative implementation of approximate dynamic 
programming, and by permitting a small compromise in optimality, the computational complexity of 
finding a (sub)optimal solution may be significantly reduced. For SLS, efficient solution algorithms (the 
so-called relaxed switched Riccati iterations) exist that find the optimal controllers for fairly large dimen- 
sional discrete-time systems. The reduction in computational complexity suggests that the (iterative) 
algorithms are amenable to practical implementation [41,42]. A relationship between stabilization and 
the optimal control of an SLS is established in [43]. 


31.8 Appendix A: Modeling with Autonomous Switches 


Four quantities are essential to describe the evolution of a system subject to autonomous and con- 
trolled switches: (1) the usual continuous state x(t) € IR”; (2) the usual continuous control input 
u(t) € R™; (3) a switching (discrete) control input, v(t) € Dy = {1,2,...,d,}; and (4) a discrete state 
& € D; = {1,2,...,d¢} that identifies autonomous switches. This chapter only considers autonomous 
switches that depend on the continuous state x(t) and the continuous input u(t) but do not depend on 
the current values of the discrete states, &(t) or v(t). Such systems are usually called memoryless systems. 
The evolution of the discrete state, &(f), of the memoryless system is defined by a piecewise continuous* 
function 7 : R” x R™ — De, such that 


&(t*) = (x(t), u(t)) € De (31.41) 
Thus, for each i € Dz, we define 


M; = {(x, u) € R” x R™|y(x, uw) = i} CR” x R™ 


* By piecewise continuous we mean a function that is continuous everywhere except on a finite union of switching surfaces 


that are smooth submanifolds of R” x R™ with measure 0 where it undergoes discontinuous jumps, but has well-defined 
limits in all directions. 
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M; C R" x R” is the set of pairs (x, u) corresponding to the discrete state i ¢ Dg. Let fi;,): Mi > R", 
i € Dz, j € D,, be acollection of C! vector fields associated with a system. The evolution of the continuous 
state x(t) is then described by 


(t) = fincx(s),u()),v()) (x(t), u(t), x(to) = Xo. (31.42) 


At each t > fo and for each discrete state §(t) € D:, the switching control input v(t) € D, thus selects 
the particular vector field that governs the evolution of the continuous state. 

As mentioned in Section 31.3, we assume that the continuous control input u(t) € Q, a convex and 
compact set in R”, and that the switching control input v(t) and the continuous control input u(t) are 
both measurable functions. Note that we restrict our attention to time-invariant systems, but the results 
can be easily generalized to time-varying systems. 

Given that the discrete state &(t) is completely determined by x(t) and u(t) through Equation 31.41, 
we can define for each j € D, a piecewise C! vector field:* 


filoc(t), u(t)) = fincxce),u(e)),j) (<(t), u(t) (31.43) 


and rewrite Equation 31.42 in a more convenient form: 


x(t) =fuy (x(t), u(t), x(to) = xo (31.44) 


This means the vector fields fj all have the same set of points of discontinuity. We thus refer to the 
systems described by Equations 31.42 and 31.44 as systems with decoupled switches. 

We are interested in computing optimal control laws for the system described by Equation 31.42 
or Equation 31.44. If the system only undergoes autonomous switches (D, = {1}), only the continuous 
input u(t) needs to be computed. This suggests that the complexity of the optimal control problem might 
not be any different than in the traditional case. In contrast, for systems with controlled switches we 
need to compute the sequence of switching times f),..., fn (including n), the sequence of discrete inputs 
V1,---+5Vy, as well as the continuous input u(t) on each interval [¢;, t; + 1] for i=0,...,n—1. It would 
therefore appear that for systems with controlled switches the optimal control problem has combinatorial 
complexity. As discussed in this chapter, both these cases have the same complexity and are amenable 
to traditional nonlinear programming techniques such as sequential quadratic programming (SQP). This 
further implies that for the systems with memoryless autonomous switches and controlled switches the 
optimal control problem is no more complex than the traditional smooth problem. 


31.9 Appendix B: Numerical Solution Using Direct Collocation 


This section overviews the direct numerical solution algorithm and strategy used in the solution of the 
unicycle example without first having to apply the necessary conditions for optimality. Specifically, we 
discuss the collocation method for solving hybrid optimal control problems [21,44-47]. 

Given the embedded PI and the state equation and constraints mentioned in Section 31.3, one dis- 
cretizes these equations using the collocation method. These discretized equations convert the EOCP 
into a finite-dimensional nonlinear programming problem (NLP), where states and inputs are treated as 
unknown variables. The NLP can be solved using an SQP solver, such as fmincon in the optimization 
toolbox of MATLAB®. The discretization-and-collocation technique consists of several steps that have 
two main stages: (1) time discretization, and state and input function approximations by a finite number 


* Similarly as before, by piecewise C!, we mean a function that is C! everywhere, except on a finite union of switching 


surfaces that are smooth submanifolds R” x R™ with measure 0 where the function is not differentiable and undergoes 
discontinuous jumps, but has well defined limits in all directions. 
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of polynomial basis functions; (2) approximation of the continuous state dynamics and PI integrand by 
discrete-state and discrete-input-dependent counterparts. 

Without going through a lengthy derivation, the continuous-time interval [fo, ty] is discretized into 
a sequence of points to < t) < t) <--- < ty_-1 < ty =T, where, for simplicity, we take t; — §_1 = h, for 
j=l,...,N. A “hat” notation is also used to distinguish the numerically estimated state and control 
values from their actual counterparts that are “hatless,” for example, %; = x(t), Ho j = Ho(t)), M1; = a(t), 
and 9; = 0(f;). The collocation method used here assumes triangular basis functions for the state and 
piecewise constant basis functions (derivatives of triangular functions) for the controls. Specifically, the 
estimated state is given by 
N 
R(t) = D> R(t) (31.45) 
j=0 


where the X;’s are to be determined and the triangular basis functions are given by 


t— tj—-1 


5 fj-1 <t< tj 
g(t) =} f+1—-Ft (31.46) 
rr a tj <t< bit 


0, elsewhere 


We note two points: the method is not restricted to using triangular basis functions and each of the 
gj(t)’s is a time shift of the previous one. 

As summarized in [45], the theoretical approach for computing the controls is to extend the state 
space with new state variables, xext € R™+1, whose derivative are the desired controls, u(t) € R™ and 
v(t) € [0,1] C R, to be computed. However, our choice of triangular basis functions for the states renders 
the control inputs piecewise constant and we simply solve directly for these (constant) control values. 
Specifically, the estimates of the control inputs are given by 


N 
ity) fa], 
FA 7 » H y(t) (31.47) 


where the piecewise constant basis functions are given by 


1 f1<t<fj 


a {, elsewhere oe 


Here we note that by the definition of the basis functions in Equation 31.48, the control values computed 
at tj are enforced over the interval h-1<t<t. 

The essence of the midpoint rule in the collocation method is to enforce the constraints at the midpoints 
of each interval [t-1, tj] for j = 1,...,N. There results the discretized embedded state dynamics 


a . Sith. . Kort 
Xj = Xj-1 th-(1 i) ( é D | i) t h-¥iefi (oa) (31.49) 


for j=1,...,N, with fo(-) and f;(-) the discretized state dynamics in modes —0 and —1, respectively. 
Thus the solution to the EOCP is given by the following NLP: Minimize 
a 
J = an (tan) + D7 Sh [Fe (tj ¥ taj tay, Bj) + Fe (th1, 5-1, ho, thy, Hj Bi) } (31.50) 
j=l 

over the controls (aj, ¥;) € Q x [0,1], subject to Equation 31.49 and all other equality constraints rep- 
resented as g(Xj_1,%;, ij, ¥j, pj) =0. Here Fg(-) is the integrand of the PI properly discretized and ); 
represents various intermediate constraints. 
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32.1 Introduction 


Methods for automatic tuning of PID controllers have been one of the results of the active research 
on adaptive control. PID controllers are defined in Section 9.5. A result of this development is that the 
design of PID controllers is going through a very interesting phase. Practically, all PID controllers that 
are designed now have at least some features for automatic tuning. Automatic tuning has also made it 
possible to generate automatically gain schedules. Many controllers also have adaptation of feedback and 
feedforward gains. Overviews of the development are given in [1,2]. 

The most important component of the adaptive controllers and automatic tuning procedures is the 
design method. The next section presents some of the most common design methods for PID con- 
trollers. These design methods are divided into three categories: (1) future-based techniques, (2) analytical 
methods, and (3) methods that are based on optimization. 

Section 32.3 treats adaptive techniques. An overview of different uses of these techniques is first 
presented, followed by a more detailed treatment of automatic tuning, gain scheduling, and adaptive 
control. Section 32.4 gives an overview of how the adaptive techniques have been used in commercial 
controllers. References are at the end of the chapter. 


32.2 Design Methods 


To obtain rational methods for designing controllers it is necessary to deal with specifications and models. 
In the classical Ziegler-Nichols methods, the process dynamics are characterized by two parameters, a gain 
and a time. Another approach is used in the analytical design methods, where the controller parameters 
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are obtained from the specifications and the process transfer function by a direct calculation. Optimization 
methods allow for compromise between several different criteria. These approaches are discussed here. 


32.2.1 Specifications 


When solving a control problem, it is necessary to understand the primary goal of control. Two common 
control objectives are to follow the setpoint and to reject disturbances. It is also important to have an 
assessment of the major limitations, which can be system dynamics, nonlinearities, disturbances, or pro- 
cess uncertainty. Typical specifications on a control system may include attenuation of load disturbances, 
setpoint following, robustness to model uncertainty, and lack of sensitivity to measurement noise. 


32.2.1.1 Attenuation of Load Disturbances 


Attenuation of load disturbances is of primary concern for process control. The disturbances may enter 
the system in many different ways, but it is often assumed that they enter at the process input. Let e be the 
error caused by a unit step load disturbance at the process input. Typical quantities used to characterize 
the error are maximum error, time to reach maximum, settling time, decay ratio, and the integrated 
absolute error (IAE) which is defined by 


IAE= 1 le(t)|dt. (32.1) 
0 


32.2.1.2 Setpoint Following 


Setpoint following is of primary interest in motion control, but of less importance for process control 
because production rates are not changed so often. Furthermore, the response to setpoint changes can 
be improved by setpoint weighting or by prefiltering of the command signal. Specifications on setpoint 
following may include requirements on rise time, settling time, decay ratio, overshoot, and steady-state 
offset for step and ramp changes in setpoint. 


32.2.1.3 Robustness to Model Uncertainty 


It is important that the controller parameters are chosen in such a way that the closed-loop system is 
not too sensitive to changes in process dynamics. There are many ways to specify the sensitivity. Many 
different criteria are conveniently expressed in terms of the Nyquist plot of the loop transfer function, and 
its distance to the critical point —1. The gain and phase margins are classical robustness measures, see 
Section 9.1. A drawback is that even if both are specified the Nyquist curve can still be close to the critical 
point. Maximum sensitivity M, is a better robustness measure since 1/M, is the shortest distance between 
the Nyquist plot and the critical point. A requirement on M; simultaneously captures requirements on 
both gain and phase margins because of the following inequalities: 


M; 
M,—1° 


8m = 


1 
> 2arcsin ( ): 
bi ce meice aaa ST 


32.2.1.4 Sensitivity to Measurement Noise 


Care should always be taken to reduce measurement noise by appropriate filtering, since it will be fed 
into the system through the feedback. It will generate control actions and control errors. Measurement 
noise is typically of high frequency. The high-frequency gain of a PID controller is 


Ky =K(L+N), 


where K is the controller gain and N is the derivative gain limitation factor. See Section 9.5. Note that 
N = 0 corresponds to PI control, and N — oo corresponds to PID control without filtering and with 


Automatic Tuning of PID Controllers 32-3 


FIGURE 32.1 Determining a first-order plus dead-time model from a step response. Time constant T can be 
obtained either as the distance AB or the distance AC. 


infinite high-frequency gain. Multiplication of the measurement noise by Kyy gives the fluctuations in 
the control signal that are caused by the measurement noise. Also note that there may be a significant 
difference in Kyr for PI and PID control. It is typically an order of magnitude larger for a PID controller, 
since the gain normally is higher for a PID controller than for a PI controller, and N is typically around 10. 
It could also be advantageous to use a second-order filter as discussed in Section 9.1. 


32.2.2 Feature-Based Techniques 


The simplest design methods are based on a few features of the process dynamics that are easy to obtain 
experimentally. Typical time-domain features are static gain Kp, dominant time constant T, and dominant 
dead time L. They can all be determined from a step response of the process, see Figure 32.1. Static gain Kp, 
dominant time constant T, and dominant dead time L can be used to obtain an approximate first-order 
plus dead-time model for the process as given in Equation 32.2. 


K 

P —sL 
G,(s) = 32.2 
p(s) l+sT° G2) 


Typical frequency-domain features are static gain Kp, ultimate gain K,, and ultimate period T,,. They 
are defined in Figure 32.2. 


Im G(i@) 


Ultimate point @=0 


Re G(i@) 


FIGURE 32.2 Static gain Kp, ultimate gain K,,, and ultimate period T,, defined in the Nyquist diagram. Static gain 
Kp is the point on the Nyquist plot at w = 0. Ultimate gain K,, is —1 divided by the ultimate point. Ultimate period 
T,, is 2m divided by the frequency corresponding to the ultimate point. 


32-4 Control System Advanced Methods 


TABLE 32.1 Controller Parameters Obtained from the Ziegler—-Nichols 


Step Response Method 

Controller K Tj Tq 
P 1/a 

PI 0.9/a 3L 

PID 1.2/a 2L L/2 


32.2.2.1 Ziegler-Nichols Methods 


In 1942, Ziegler and Nichols presented two design methods for PID controllers: time-domain method and 
frequency-domain method [3]. The methods are based on determination of process dynamics in terms 
of only two parameters, gain and time. The controller parameters are then expressed in terms of these 
parameters by simple formulas. In both methods, the design specification of quarter amplitude decay 
ratio was used. The decay ratio is the ratio between two consecutive maxima of the control error after a 
step change in setpoint or load. 

The time-domain method is based on a registration of the open-loop step response of the process. 
Ziegler and Nichols have given PID parameters directly as functions of a and L, defined in Figure 32.1. 
These are given in Table 32.1. 

The second method presented by Ziegler and Nichols is based on the frequency response of the process. 
They have given simple formulas for the parameters of the controller in terms of ultimate gain K, and 
ultimate period T,. These parameters can be determined in the following way. Connect a controller to 
the process, set the parameters so that control action is proportional, that is, T; = oo and Ty = 0. Increase 
the gain slowly until the process starts to oscillate. The gain when this occurs is K, and the period of the 
oscillation is T,,. The parameters can also be determined approximately by relay feedback as is discussed 
in Section 32.3. The controller parameters are given in Table 32.2. 


32.2.2.2 Modifications of the Ziegler-Nichols Methods 


The Ziegler-Nichols methods do not give satisfactory control. Therefore, there have been many modifi- 
cations of the method [2,4-6]. The reason is that they give closed-loop systems with very poor damping. 
The design criterion “quarter amplitude decay ratio” corresponds to a relative damping of ¢ ~ 0.2 which 
is much too small for most applications. The maximum sensitivity is also much too large, which means 
that the closed-loop systems obtained are too sensitive to parameter variations. 

The Ziegler-Nichols methods do, however, have the advantage of being very easy to use. Many efforts 
have therefore been made to obtain tuning methods that retain the simplicity of the Ziegler-Nichols 
methods but give improved robustness. 


32.2.2.3 AMIGO Tuning 


Significantly better tuning rules can be obtained if the process dynamics are described in terms of three 
parameters instead of two. An early step in this direction was made by Cohen and Coon, who assumed 


TABLE 32.2 Controller Parameters Obtained from the Ziegler—-Nichols 
Frequency Response Method 


Controller K Tj Tq 
P 0.5Ky 
PI 0.4Ky 0.8Ty 


PID 0.6Ky 0.5Ty 0.12Ty 
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that the process was given by Equation 32.2, which has three parameters [5]. Their design did, however, 
also give very sensitive systems. 

The AMIGO (Approximate M-constraint Integral Gain Optimization) tuning rules [2] where derived 
in the following way. PI and PID controllers were designed for a large test batch consisting of 134 process 
models. The design goal was to maximize the controller integral gain subject to robustness constraints 
expressed by maximum sensitivities (MIGO). The process models were then approximated by simple 
models and relations between model parameters and controller parameters were derived. These relations 
form the AMIGO (Approximate MIGO) tuning rules. Two versions are available, one step response 
method and one frequency response method. 

PID controllers were designed by maximizing integral gain subject to the constrains that the sensitivity 
functions M, and M; should be smaller than 1.4. In the step response method it was attempted to correlate 
controller gains to parameters of the simple FOTD (First order plus time delay) model (Equation 32.2). 
The parameters describing the process were obtained by approximating the step responses by Equation 
32.2. Processes with integration are approximated by 


K, 
Gp(s) = er", (32.3) 


where K, is the velocity gain and L the time delay. 

Figure 32.3 illustrates the relations between the PI controller parameters and the process parameters 
for all processes in the test batch. The controller gain is normalized by multiplying it either with the static 
process gain Kp or with the parameter a = KpL/T = K,L. The integral time is normalized by dividing 
it by T or by L. The controller parameters in Figure 32.3 are plotted versus the normalized dead time 
t=L/(L+T). The figure shows that there is a good correlation between the normalized controller 
parameters and normalized dead time. 

The solid lines in Figure 32.3 correspond to the AMIGO tuning formula, 


c= 98 5 (035 LF ) T 

ae peal | ere een 

P ( aS (32.4) 
13LT? 

T; = 0.35L4 


T2+12LT +712’ 


and the dotted lines show the limits for 15 % variations in the controller parameters. Almost all processes 
included in the test batch fall within these limits. 

For integrating processes, K, and T go to infinity and K,/T = Ky. Therefore, the AMIGO tuning rules 
(Equation 32.4) can be simplified to 


0.35 
K=-—, 

K,L (32.5) 
T; = 13.41. 


for integrating processes. 
The suggested AMIGO tuning rules for PID controllers are 


1 T 
0.4L + 0.8T 
seman ee RG 
0.5LT 


= O3LaT 


(32.6) 


> 


Tq 
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aK vs.T 
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FIGURE 32.3 Normalized PI controller parameters plotted versus normalized time delay t. The solid lines corre- 
spond to the AMIGO design rule (Equation 32.4), and the dotted lines indicate 15 % parameter variations. 


For integrating processes, Equation 32.6 can be written as 
K =0.45/(K,L), 
Tj = 8L, 
Tg = 0.5L. 


(32.7) 


In the AMIGO frequency-response method, the processes in the test batch are characterized by the 
three parameters static gain Kp, ultimate gain K,, and ultimate period T,,. The processes are classified 
according to the gain ratio k = 1/(KpK,). The AMIGO tuning rules are appropriate for processes where 


K > 0.2. The PI tuning rules are 


K =0.16K,, 
ol 
~ 14+4.5« 


i u 


and the PID rules are 


K = (0.3—0.1k*)K,, 
_ 0.6 
~ 142k 
0.15(1 — 
pea 
1—0.95k 


Ty, 


i 


ue 


(32.8) 


(32.9) 
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32.2.3 Tuning Based on Gain and Phase Margins 


Figure 32.3 shows that integral time is close to the process time constant for a reasonably wide range of 
t. We can use this insight to obtain a Ziegler—Nichols like formula which contains specifications on gain 
and phase margins. A PI controller that cancels the process pole is, 


K(1+sT) 
C(s) = ———.. 
(s) 7 
The corresponding loop transfer function is 
KK, 
P i—sL 
G; = ——e™, 
ery | 


and the gain crossover frequency is wg. = KpK /T. The phase at the gain crossover frequency is 


; Tt m KpKL 
arg Gi(iM@ge) = 5 Ogcel = a 7 
Requiring a phase margin @» gives 
Qn T 
on aaa Om = > — Pm: 


With @m = 1/3 we obtain a, = 1/6 = 0.52 © 0.5. The tuning rule becomes 


_ mT KK Om 


=o3 =p Er (32.10) 
p p 


Note that k;KpL = Oy. 

The control law (Equation 32.10) gives good control when L © T, but not when L >> T or L<< T. 
The proportional gain goes to zero with L/T and the gain is too low for delay-dominated processes. To 
explore how proportional gain should be increased we approximate the transfer function by 


P(s) © Kye", 
and we find that the proportional gain can be increased to 
1 


K < —, 
&mKp 


where gj, is the gain margin. A gain margin g,, = 5 gives K < 0.2/Kpy. The formula (Equation 32.10) gives 
very poor damping for systems with lag-dominated dynamics, which can be approximated by 


K, 

P o—sL 
P y — u 
(s) ST. 


The gain crossover frequency for a proportional controller that gives the phase margin ~ is then 


Om 
re 


and the controller has the gain 
Onl 
KL 
Note that this is the same gain as was obtained for systems with balanced dynamics, compare with 
Equation 32.10. Adding integral action reduces the phase margin. To avoid reducing it too much we 
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require that @,g-T; > B, which corresponds to an increase of the phase lag of B, where B typically is in 
the range of 0.1-0.5. The condition can also be written as Tj > BL. The corresponding condition on 
integral gain is 

K nT 
=—_< “ 

T; ~ BKpL? 


i 


Summarizing, we find the following tuning rule: 


mT L 
KpL for T < Omm> 
K = 
1 f L = 
— Or — > OmSm> 
T 
pn (32.11) 
mT L o1 
for —<-, 
BKpL? T 8B 
ki= Qin Ee8 1 
for — > -, 
KpL TB 


where Qn = 1/2 — Pm, and @» is the phase margin in radians and g,, is the gain margin. 


32.2.4 Analytical Methods 


If the process can be described well by a simple model, the controller parameters can be obtained by a 
direct calculation. This approach is treated in this section. 


32.2.4.1 Pole Placement 


If the process is described by a low-order transfer function, a complete pole-placement design can be 
performed. A PI controller has two parameters and the two poles can be placed and a PID controller can 
place three poles. The zeros can be influenced by using setpoint weighting. Equations for the controller 
parameters are given in [2]. To have a robust system the desired closed loop poles should be chosen with 
care [7]. 


32.2.4.2 \-Tuning 


Let Gp and G; be the transfer functions of the process and the controller. The closed-loop transfer function 
obtained with error feedback is then 


GpG 
Go = 2. 
1+G)G, 
Solving this equation for G, gives 
1 G 
C= ees (32.12) 
Gp 1—Go 


If the closed-loop transfer function Go is specified and G, is known, it is thus easy to compute G,. 
The method, called \-tuning, was developed for processes with long dead time L [8]. Consider a process 
with the transfer function 


Gp = — ee (32.13) 
Assume that the desired closed-loop transfer function is specified as 
eh 
Go = Thar’ (32.14) 


where i is a tuning parameter. The time constants of the open- and closed-loop systems are the same 
when ) = 1. The closed-loop system responds faster than the open-loop system if \ < 1. Itis slower when 
x> 1. 
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It follows from Equation 32.12 that the controller transfer function becomes 


= 1+sT 
~ Kp(1 + XsT — ee)’ 


Cc 


When L = 0 this becomes a PI controller with gain K = 1/().Kp) and integral time T; = T. The sensitivity 
function obtained with -tuning is given by 


et 2 1+sxT — ee 
1+s\T  14+s5XT 


S(s) =1 


The maximum sensitivity M, is always less than 2 if the model is correct. With unmodeled dynamics, 
the sensitivity may be larger. The parameter \ should be small to give a low IAE, but a small value of 
increases the sensitivity. 


32.2.4.3 Internal Model Control 


The internal model principle is a general method for design of control systems that can be applied to PID 
control [9]. A block diagram of such a system is shown in Figure 32.4. It is assumed that all disturbances 
acting on the process are reduced to an equivalent disturbance d at the process output. In this figure, G,y, 
denotes a model of the process, Gi, is an approximate inverse of Gj, and G is a low-pass filter. The name 
internal model controller derives from the fact that the controller contains a model of the process. This 
model is connected in parallel with the process. 

If the model matches the process, that is, Gj, = Gp, the signal e is equal to the disturbance d for all 
control signals u. If G =1 and Gi, is an exact inverse of the process, then the disturbance d will be 
canceled perfectly. The filter G is introduced to obtain a system that is less sensitive to modeling errors, 
and to ensure that the system GG, is realizable. A common choice is G(s) = 1/(1+s¥), where F is a 
design parameter. 

The controller obtained by the internal model principle can be represented as an ordinary series 
controller with the transfer function 

GGT, 


<= —__. (32.15) 
1—G/GhGn 


From this expression it follows that controllers of this type cancel process poles and zeros. The controller 
is normally of high order. Using simple models it is, however, possible to obtain PI or PID controllers. 
To see this, consider a process with the transfer function 


{Controller 


FIGURE 32.4 Block diagram of a closed-loop system with a controller based on the internal model principle. 
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An approximate inverse, where no attempt is made to find an inverse of the time delay, is given by 


1+sT 
Gi (s) = a ; 
Kp 
Choosing the filter 
G — 
Ps £5 


and approximating the time delay by 


eh 1 —sL 


Equation 32.15 now gives 
1+sT 


G,(s) = Kps(h+ f) 


which is a PI controller. If the time delay is approximated instead by a first-order Padé approximation 


Sly 1—sL/2 
14+sL/2° 


Equation 32.15 gives instead the PID controller 


— (l+sl/2)1+sT) — (+sL/2)(1+sT) 


Gl) = L4 F+SFL/D © Ks +F) 


An interesting feature of the internal model controller is that robustness is considered explicitly in the 
design. Robustness can be adjusted by selecting the filter Gr properly. A trade-off between performance 
and robustness can be made by using the filter constant as a design parameter. 

The internal model control (IMC) method can be designed to give excellent responses to setpoint 
changes. Since the design method inherently implies that poles and zeros of the plant are canceled, the 
response to load disturbances may be poor if the canceled poles are slow in comparison with the dominant 
poles. This is discussed in the next section. 


32.2.4.4 Skogestad’s Internal Model Controller 


Skogestad has developed a simple tuning method based on internal model control for FOTD systems. 
The closed loop transfer function is specified as 


1 
G = gh 
ica cole ae sT oI 


For an FOTD system it then follows from Equation 32.13 that the controller transfer function is 


1+sT —  itsT 
Kp(1+sTq—e7!) — sKp(Ta +L)’ 


G-(s) = 


where the exponential function is approximated using a Taylor-series expansion. The closed-loop 
response time is specified to be proportional to the time delay L and the integral time is modified 
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for lag-dominated processes leading to the following tuning rule for PI control: 


E 


> 


2KpL (32.16) 
T; = min (T, 8L). 


Note that the proportional gain has the same form as for the Ziegler—Nicols rule but that the gain is 
smaller. Also note the similarity with Equation 32.11. 

The same parameters are used for a PID controller in series form, and the derivative time is chosen as 
the shortest time constant. 


32.2.5 Loop-Shaping 


Most traditional control design techniques can also be applied to PID control. Loop-shaping tries to 
achieve a desired loop transfer function by suitable choice of the controller. The following procedures are 
minor modifications of techniques proposed in [10]. 

We start by choosing a desired gain crossover frequency ™g-. The choice could be governed by the 
requirement of attenuation of load disturbances. Using a controller with integral action the attenuation 
of a sinusoidal load disturbances with frequency wg is approximately wg: /@ . This approximation is 
particularly good when the phase margin is 60° because the sensitivity crossover frequency are then equal 
to the gain crossover frequency. To have the phase margin )m we obtain the following condition: 


LC(iedge) + ZP(iege) = —180° + bins (32.17) 


where $ is the required phase margin. 

Any stable process can be controlled by an integrating controller. Since an integrating controller has 
a phase lag of 90° we find that an integral controller can be used if the phase lag of the process is in the 
range of 0-90° — $,, (0-30° with a phase margin $,, = 60°). The integral gain is then given by 


Dec 
a 
" [P(iadgc)| 


PI control can be used if higher gain crossover frequencies are desired. Since a PI controller has a phase 
lag between 0° and 90° it follows from Equation 32.17 that PI control can be used if the phase lag of 
the process is in the range of 90° — om (30-120° with a phase margin ,, = 60°). The phase lag of a PI 
controller at gc is 90 — arctan wg, T;. Equation 32.17 then gives 


Wec Ti = —ZP(imgc) — 90° + bn. 


The integral gain is then given by 
ki —_— 


IGling)ly/1 + 02.7? 


A PID controller can provide lead and it is then possible to choose even higher gain crossover fre- 
quencies. The phase lead is at most 90° which corresponds to pure derivative control, and the crossover 
frequency can be such that the process phase lag approaches 270° — On. 

The limits of 270° — @,, phase lag for PID control and 180° — 6, for PI control are too optimistic 
because they correspond to pure derivative and pure proportional control. In practice, the allowable 
phase lags are smaller because integral action is needed. 
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To find a PID controller of the form 


1+sT;+s*T;T 
ee eae 
s(1+sTy) 


(32.18) 


we first allocate a phase lag to the filter by picking the filter time constant Ty so that the phase os = 
arctan Wg Ty has a reasonable value, typically about 10°. The required phase advance is then 


b= —ZP(idgc) ar bf +Om— 90°. 
This phase lag has to be provided by the numerator 1 + sT; + s?Tq of the transfer function 32.18, hence 


gc Ti _ Age T; 
2. 7: — va 
1— 09. TiTa a — wT; 


tanod = 


Since there are two parameters T; and Ty, one of them has been fixed by choosing a = T;/Ty as a 
tuning parameter which will be adjusted later (typical values are in the range of 0.1-1). Solving the above 
equation for T; gives 

—at+ Ja? + 40 tan? > 


for 6 < 90°, 
t= 2wgetand (32.19) 
—a— Va? + 4a tan? > 
for 90° <  < 180°. 
2Wgc tan 


Having determined T; integral gain is then adjusted to give unit gain at the crossover frequency; hence 


2 
Oge,/1+ Weel; 


© [P(iedge)(1 — 12, T? + i@ge Ti)| 


(32.20) 


i 


The design parameter a can be adjusted, for example, to maximize the integral gain. An interactive 
learning module for loop-shaping can be downloaded from http://www.calerga.com/contrib/index.html. 


32.2.6 Optimization Methods 


Optimization is a powerful tool for design of controllers. The method is conceptually simple. A controller 
structure with a few parameters is specified. Specifications are expressed as inequalities of functions of the 
parameters. The specification that is most important is chosen as the function to optimize. The method 
is well suited for PID controllers where the controller structure and the parameterization are given. 
There are several pitfalls when using optimization. Care must be exercised when formulating criteria and 
constraints; otherwise, a criterion will indeed be optimal, but the controller may still be unsuitable because 
of a neglected constraint. Another difficulty is that the loss function may have many local minima. A third 
is that the computations required may be excessive. Numerical problems may also arise. Nevertheless, 
optimization is a good tool that has successfully been used to design PID controllers. 

Popular optimization criteria are the IAE, the integrated time absolute error (ITAE), and the integrated 
square error (ISE). They are mostly done for the first-order plus dead-time model as given in Equation 32.2. 
Many tables that provide controller parameters based on optimization have been published [6]. A very 
general optimization algorithm which considers many different constrains is presented in [11]. 


32.2.6.1 Modulus Optimum and Symmetrical Optimum 


Modulus Optimum (BO) and Symmetrical Optimum (SO) are two design methods that are based on 
optimization. The acronyms BO and SO are derived from the German words Betrags Optimum and 
Symmetrische Optimum. These methods are based on the idea of finding a controller that makes the 
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frequency response from setpoint to plant output as close to one as possible for low frequencies. If G(s) 
is the transfer function from the setpoint to the output, the controller is determined in such a way that 
G(0) = 1 and that d"|G(iw)|/dw” = 0 at w = 0 for as many nas possible. 

If the closed-loop system is given by 


2 
Wo 


G(s) = ——__——_., 
32+ /2wos + oF 


the first three derivatives of |G(iq)| will vanish at the origin. If the transfer function G in the example is 
obtained by error feedback of a system with the loop transfer function Ggo, the loop transfer function is 
G(s) ws 
Ggo(s) = = ; 
1—G(s)_— s(s+ V2) 
which is the desired loop transfer function for the method called modulus optimum. 
If the closed-loop transfer function is given by 


3 
Wo 


~ (s+ao)(s? + aos + w2) 


G(s) (32.21) 
the first five derivatives of |G(io)| will vanish at the origin. A system with this closed-loop transfer 


function can be obtained with a system having error feedback and the loop transfer function 
3 
0 


Ge(s) = . 
es) s(s? + 2a9s + 20) 


The closed-loop transfer function (Equation 32.21) can also be obtained from other loop transfer functions 
if a two-degree of freedom controller is used. For example, if a process with the transfer function 


2 
Wo 


oO s(s + 209) 

is controlled by a PI controller having parameters K = 2, T; = 2/9, and b = 0, the loop transfer function 
becomes 

we (2s + wo) 
s2(s+ 2a)” 
The symmetric optimum aims at obtaining the loop transfer function given by Equation 32.22. Note 
that the Bode diagram of this transfer function is symmetrical around the frequency w = wo. This is the 
motivation for the name symmetrical optimum. 

The methods BO and SO can be called loop-shaping methods since both methods try to obtain a 
specific loop transfer function. The design methods can be described as follows. It is first established 
which of the transfer functions, Ggo or Gso, is most appropriate. The transfer function of the controller 
G,(s) is then chosen so that Ge(s) = G,(s)Gp(s), where Gy is the chosen loop transfer function. 


Gso = (32.22) 


32.2.6.2 Cancellation of Process Poles 


The PID controller has two zeros. Many design methods choose these zeros so that they cancel one or two 
of the dominant process poles. This often results in a simple design as in IMC or SO optimization. The 
response to setpoint changes is generally good. The responses to load disturbances may, however, be poor 
if the canceled poles are slower than the dominating closed-loop poles. Figure 32.5 illustrates the problem. 
A process with a time constant T = 10 is controlled with a PI controller with integral time Tj = 10 anda 
suitable controller gain. The design is made so that the closed-loop time constant is significantly shorter 
than the open-loop time constant. This is seen in the response to the setpoint change. However, the load 
disturbance response is very sluggish since the open-loop time constant is present and dominating. The 
figure also shows a retuned controller, where pole cancellation is avoided. 
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FIGURE 32.5 Simulation of a closed-loop system obtained by pole cancelation. The process transfer function is 
G(s) = e~*/(10s + 1), and the controller parameters are K = 6.67 and T; = 10. The upper diagram shows setpoint 
Ysp = 1 and process output y, and the lower diagram shows control signal u. The figure also shows the responses to a 
retuned controller with K = 6.67, T; = 3 and b= 0.5. 


32.3 Adaptive Techniques 


This section gives an overview of adaptive techniques. It starts with a discussion of uses of the different 
techniques, followed by a more detailed presentation of automatic tuning, gain scheduling, and adaptive 
control. The section ends with a presentation of how the adaptive techniques have been used in industrial 
controllers. 


32.3.1 Use of the Adaptive Techniques 


The word adaptive techniques is used to cover autotuning, gain scheduling, and adaptation. Although 
research on adaptive techniques has almost exclusively focused on adaptation, experience has shown 
that autotuning and gain scheduling have much wider industrial applicability. Figure 32.6 illustrates the 


appropriate use of the different techniques. 
Constant but Predictable Unpredictable 
unknown dynamics changes in dynamics changes in dynamics 


ae 


Gain scheduling 
Constant controller Predictable Unprdictable 
parameters parameter changes parameter changes 


FIGURE 32.6 When to use different adaptive techniques. 
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Controller performance is the first issue to consider. If requirements are modest, a controller with 
constant parameters and conservative tuning can be used. Other solutions should be considered when 
higher performance is required. 

If the process dynamics are constant, a controller with constant parameters should be used. The 
parameters of the controller can be obtained by autotuning. 

If the process dynamics or the character of the disturbances are changing it is useful to compensate 
for these changes by changing the controller. If the variations can be predicted from measured signals, 
gain scheduling should be used since it is simpler and gives superior and more robust performance than 
continuous adaptation. Typical examples are variations caused by nonlinearities in the control loop. 
Autotuning can be used to build up the gain schedules automatically. 

There are also cases where the variations in process dynamics are not predictable. Typical examples 
are changes due to unmeasurable variations in raw material, wear, fouling, and so on. These variations 
cannot be handled by gain scheduling but must be dealt with by adaptation. An autotuning procedure is 
often used to initialize the adaptive controller. It is then sometimes called pretuning or initial tuning. 


32.3.2 Automatic Tuning 


Automatic tuning (or autotuning) is a method where a controller is tuned automatically on demand from 
the user. Typically, the user will either push a button or send a command to the controller. Automatic 
tuning is sometimes called tuning on demand or one-shot tuning. 

Automatic tuning can also be performed by external devices that are connected to the control loop 
during the tuning phase. Since these devices are supposed to work together with controllers from 
different manufacturers, they must be provided with quite a lot of information about the controller 
structure and parameterization in order to provide appropriate controller parameters. Such information 
includes controller structure (series or parallel form), sampling rate, filter time constants, and units of the 
different controller parameters (gain or proportional band, minutes or seconds, time or repeats/time). 

The automatic tuning procedures can be divided into methods that are based on step response experi- 
ments, and methods based on frequency response experiments. 


32.3.2.1 Step Response Methods 


Most methods for automatic tuning of PID controllers are based on step response experiments. When the 
operator wishes to tune the controller, an open-loop step response experiment is performed. A process 
model is then obtained from the step response, and controller parameters are determined. This is usually 
done using simple look-up tables as in the Ziegler-Nichols method. 

The greatest difficulty in carrying out the tuning automatically is in selecting the amplitude of the step. 
The user naturally wants the disturbance to be as small as possible so that the process is not disturbed 
more than necessary. On the other hand, it is easier to determine the process model if the disturbance is 
large. The result of this dilemma is usually that the operator himself has to decide how large the step in 
the control signal should be. 

Controllers with automatic tuning which are based on this technique have become very common in 
the last few years. This is especially true of temperature controllers. 


32.3.2.2 The Relay Autotuner 


Frequency-domain characteristics of the process can be determined from experiments with relay feedback 
in the following way [12]. When the controller is to be tuned, a relay with hysteresis is introduced in the 
loop, and the PID controller is temporarily disconnected; see Figure 32.7. For large classes of processes, 
relay feedback gives an oscillation with period close to the ultimate frequency w,, as shown in Figure 32.8. 
The control signal is a square wave and the process output is close to a sinusoid. The gain of the transfer 
function at this frequency is also easy to obtain from amplitude measurements. Describing function 
analysis can be used to determine the process characteristics. The describing function of a relay with 
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Process 


FIGURE 32.7 The relay autotuner. In the tuning mode, the process, is connected to relay feedback. 


hysteresis is 


4d ( €\2 € 
N@= 1 ( ) aan ie 

ma a a 
where d is the relay amplitude, « is the relay hysteresis, and a is the amplitude of the input signal. The 
negative inverse of this describing function is a straight line parallel to the real axis; see Figure 32.9. The 
oscillation corresponds to the point where the negative inverse describing function crosses the Nyquist 
curve of the process, that is, where 

G(iw) = — N@’ 
Since N(a) is known, G(iw) can be determined from the amplitude a and the frequency w of the oscillation. 

Note that the relay experiment is easily automated. There is often an initialization phase where the 

noise level in the process output is determined during a short period of time. The noise level is used 
to determine the relay hysteresis and a desired oscillation amplitude in the process output. After this 
initialization phase, the relay function is introduced. Since the amplitude of the oscillation is proportional 
to the relay output, it is easy to control it by adjusting the relay output. Also note in Figure 32.8 that a 
stable oscillation is established very quickly. The amplitude and the period can be determined after about 
10 s only. The average residence time of the system is 12 s, which means that it would take about 40 s for 
an open-loop response to reach steady state. 


32.3.3 Gain Scheduling 


By gain scheduling we mean a system where controller parameters are changed depending on measured 
operating conditions. The scheduling variable can, for instance, be the measurement signal, controller 


FIGURE 32.8 Relay output u (dashed) and process output y (solid) for a system under relay feedback. 
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FIGURE 32.9 The negative inverse describing function of a relay with hysteresis —1/N(a) and a Nyquist 
curve G(iw). 


output, or an external signal. For historical reasons the word “gain scheduling” is used even if other 
parameters such as derivative time or integral time are changed. Gain scheduling is a very effective way 
of controlling systems whose dynamics change with the operating conditions. 

The notion of gain scheduling was originally used for flight control systems, but it is being used 
increasingly in process control. It is, in fact, a standard ingredient in some single-loop PID controllers. 
For process control applications, significant improvements can be obtained by using just a few sets of 
controller parameters. 


32.3.4 Adaptive Control 


An adaptive controller is a controller whose parameters are continuously adjusted to accommodate 
changes in process dynamics and disturbances. Parameters can be adjusted directly or indirectly via 
estimation of process parameters. There is a large number of both direct and indirect methods available. 
Adaptation can be applied both to feedback and feedforward control parameters. Adaptive controls can 
be described conveniently in terms of the methods used for modeling and control design. 


32.3.4.1 Model-Based Methods 


All indirect systems can be represented by the block diagram in Figure 32.10. There is a parameter 
estimator that determines the parameters of the model based on observations of process inputs and 
outputs. There is also a design block that computes controller parameters from the model parameters. 
The parameters can either be estimated recursively or batchwise. 

design 


owed 


FIGURE 32.10 Block diagram of indirect systems. 


Controller 
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32.3.4.2 Rule-Based Methods 


In the direct methods, the key issues are to find suitable features that characterize relevant properties of 
the closed-loop system and appropriate ways of changing the controller parameters so that the desired 
properties are obtained. 

The majority of the PID controllers in industry are tuned manually by instrument engineers. The tuning 
is done based on past experience and heuristics. By observing the pattern of the closed-loop response to 
a setpoint change, the instrument engineers use heuristics to directly adjust the controller parameters. 
The heuristics have been captured in tuning charts that show the responses of the system for different 
parameter values. A considerable insight into controller tuning can be developed by studying such charts 
and performing simulations. The heuristic rules have also been captured in knowledge bases in the form 
of crisp or fuzzy rules. Rules of this type are used in several commercial adaptive controllers. Most 
products will wait for setpoint changes or major load disturbances. When these occur, properties such as 
damping, overshoot, period of oscillations and static gains are estimated. Based on these properties, rules 
for changing the controller parameters to meet desired specifications are executed. 


32.3.5 Adaptive Feed Forward 


Feedforward control deserves special mention. It is a very powerful method for dealing with measurable 
disturbances. Use of feedforward control requires, however, good models of process dynamics. It is 
difficult to tune feedforward control loops automatically on demand, since the operator often cannot 
manipulate the disturbance used for the feedforward control. To tune the feedforward controller it is 
therefore necessary to wait for an appropriate disturbance. Adaptation is therefore particularly useful for 
the feedforward controller. 


32.4 Some Commercial Products 


Commercial PID controllers with adaptive techniques have been available since the beginning of the 
1980s [2,13]. 

There is a distinction between temperature controllers and process controllers. Temperature controllers 
are primarily designed for temperature control, whereas process controllers are supposed to work for 
a wide range of control loops in the process industry such as flow, pressure, level, and pH control 
loops. Automatic tuning and adaptation are easier to implement in temperature controllers, since most 
temperature control loops have several common features. This is the main reason why automatic tuning 
has been introduced more rapidly in these controllers. 

The process controllers can be separate hardware boxes for single loops, or distributed control systems 
(DCS) where many loops are handled by one system. 

Since the processes that are controlled with process controllers may have large differences in their 
dynamics, tuning and adaptation becomes more difficult compared to the pure temperature control 
loops. In Table 32.3, a collection of process controllers is presented together with information about their 
adaptive techniques. 

Automatic tuning is the most common adaptive technique in the industrial products. The usefulness 
of this technique is also obvious from Figure 32.6, where it is shown that the autotuning procedures are 
used not only to tune the controller, but also to obtain a comfortable operation of the other adaptive 
techniques. Most auto-tuning procedures are based on step response analysis. 

Gain scheduling is often not available in the controllers. This is surprising, since gain scheduling is 
found to be more useful than continuous adaptation in most situations. Furthermore, the technique is 
much simpler to implement than automatic tuning or adaptation. 

It is interesting to see that many industrial adaptive controllers are rule based instead of model based. 
The research on adaptive control at universities has been almost exclusively focused on model-based 
adaptive control. 
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TABLE 32.3 Industrial Adaptive Process Controllers 


Automatic Gain Adaptive Adaptive 
Manufacturer Controller Tuning Scheduling Feedback Feedforward 
Bailey Controls CLC04 Step Yes Model based - 
Control Techniques Expert controller Ramps - Model based - 
Fisher Controls DPR900 Relay Yes - - 
DPR910 Relay Yes Model based Model based 
Foxboro Exact Step - Rule based - 
Fuji CC-S:PNA 3 Steps Yes = - 
Hartmann & Braun Protronic P Step a 7 7 
Digitric P Step - = 7 
Honeywell UDC 6000 Step Yes Rule based - 
Alfa Laval Automation ECA40 Relay Yes - - 
ECA400 Relay Yes Model based Model based 
Siemens SIPART DR22 Step Yes 7 - 
Toshiba TOSDIC-215D PRBS Yes Model based - 
EC300 PRBS Yes Model based - 
Turnbull Control Systems TCS 6355 Steps - Model based - 
Yokogawa SLPC-171,271 Step Yes Rule based . 
SLPC-181,281 Step Yes Model based - 


One of the earliest rule-based adaptive controllers is the Foxboro EXACT [14]. It was released in 1984. 
In this controller, the user specifies a maximum damping and a maximum overshoot. Whenever the 
control loop is subjected to a larger load disturbance or setpoint change, the response is investigated and 
the controller parameters are adjusted according to certain rules to meet the specifications. 

Adaptive feedforward control is seldom provided in the industrial controllers. This is surprising, since 
adaptive feedforward control is known to be of great value. Furthermore, it is easier to develop robust 
adaptive feedforward control than adaptive feedback control. 

Alfa Laval Automation’s ECA400 and Fisher Controls DPR910 are identical. This controller has auto- 
matic tuning, gain scheduling, adaptive feedback, and adaptive feedforward. The automatic tuning pro- 
cedure is based on relay feedback. The automatic tuning procedure is also used to build the gain sched- 
ule automatically, and to initiate the adaptive feedback and feedforward. In this way, there is no need 
for the user to supply any information about the process dynamics. All adaptive features can be used 
automatically. 


References 


1. O’Dwyer, A., Handbook of PLand PID Controller Tuning Rules. Imperial College Press, London, 2009. 
2. 


Astrém, KJ. and Hagglund, T., Advanced PID Control, Instrumentation, Systems, and Automation 
Society, Research Triangle Park, NC, 2005. 

3. Ziegler, J.G. and Nichols, N.B., Optimum settings for automatic controllers, Trans. ASME, 64, 759, 1942. 

4. Chien, K.L., Hrones, J.A., and Reswick, J.B., On the automatic control of generalized passive systems, 
Trans. ASME, 74, 175, 1952. 

5. Cohen, G.H. and Coon, G.A., Theoretical consideration of retarded control, Trans. ASME, 75, 827, 1953. 

6. Kaya, A. and Scheib, T.J., Tuning of PID controls of different structures, Control Eng., July, 62, 1988. 

7. Astrém, KJ. and Murray, R.M., Feedback Systems and Introduction for Scientists and Engineers. Prince- 
ton University Press, Princeton, NJ, 2008. 

8. Dahlin, E.B., Designing and tuning digital controllers, Instr. Control Syst., 42, 77, 1968. 

9. Morari, M. and Zafiriou, E., Robust Process Control, Prentice-Hall, Englewood Cliffs, NJ, 1989. 


32-20 Control System Advanced Methods 


10. Messner, W.C., Classical control revisited: Variations on a theme. Proc. 10th Int. Workshop on Advanced 
Motion Control, AMC’08. Trento, 2008, pp. 15-20. 

11. Garpinger, O. and Hagglund, T. A., software tool for robust PID design. Proc. 17th IFAC World Congress, 
Seoul, Korea, 2008. 

12. Hagglund, T. and Astrém, KJ., Industrial adaptive controllers based on frequency response techniques, 
Automatica, 27, 599, 1991. 

13. Astrém, KJ., Hagglund, T., Hang, C.C., and Ho, W.K., Automatic tuning and adaptation for PID 
controllers—A survey, Control Eng. Prac., 1(4), 699, 1993. 

14. Kraus, T.W. and Myron, T-J., Self-tuning PID controller uses pattern recognition approach, Control 
Eng., June, 106, 1984. 


David W. Clarke 
University of Oxford 


33.1 Introduction 


33 


Self-Tuning Control 


Saul 


33.2 


Bad 


33.4 


BaD 
33.6 
Soul 
33.8 
33.9 


NGG CAR INGING SoiiiGtiiasstiiiysdainenadoianaiviens 33-1 
Examples of Unknown and Time-Varying Systems 

Borie Sitiple Meh Ody o.iscscnsesoiisiesinesseransasatinss 33-3 
A Plant with an Unknown Time-Varying Gain 

Pia MOIS ccncanarnuanenmamaninns 33-7 


Transfer-Function or Difference-Equation Models « 
Incorporating Disturbances 

Recursive Prediction Error (RPE) 

PS OPS sc patvnranenarcromnivnonearnaiennas 33-10 
Forgetting Factors « Variable Forgetting Factors « 
Forgetting with Multiparameter Models 


Predtetive Modes -scccsscousuiaismuwimoagns 33-16 
Minimum-Variance (MV) Control............... 33-19 
Minimum-Variance Self-Tuning..............0. 33-21 
Pole-Placement (PP) Self-Tuning........... 33-26 
Long-Range Predictive Control ..........ccee 33-28 


33.10 The Generalized Predictive Control (GPC) 


© SGE Pati costa sessvssueyaucsawsandseseataateasaseatsateate 33-32 


33.11 Robustness of Self-Tuning Controllers ......... 33-33 
POSER VINCE 2:55 acini acsieaicibicsles ea caueitnenceumues 33-35 


Most control theory is concerned with the design of feedback systems for time-invariant plants with 
known mathematical models, e.g., in the form of given transfer functions. The assumption of constant, 
known models is not valid for many modern technical or nontechnical systems, such as: 


Robotics: Inertias as seen by the drive motors vary with the end-effector position and the load mass so 
that the dynamic model varies with the robot’s attitude. 
Chemical reactors: Transfer functions vary according to the mix of reagents and catalyst in the vessel 


and change as the reaction progresses. 


People: The gain of the function {injected anaesthetic - unconsciousness} depends on the patient’s 


metabolism. 


So how can controllers be designed to cope with these initial model uncertainties and time variations 
in dynamics? One approach is to design a robust fixed controller that ensures stability for all possible 
plant dynamics, but this is often at the expense of “detuned” behavior. Instead we can embed algorithms 
inside computers that “learn from experience” and self-tune the controllers so as to improve closed- 
loop performance. Often this learning process builds up a mathematical model based on experimental 
input/output data; this operation is known as system identification or parameter estimation. The model 
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FIGURE 33.1 Structure of a self-tuning controller. 
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could be a complete transfer function or simply the gain and phase of the plant at a given input frequency. 
In the design of “learning” controllers there are two main themes: 


¢ For controlling systems that have unknown but constant dynamics, using single-shot methods 
when the controller is first commissioned: If the target control law is the “industry-standard” PID 
(proportional-integral-derivative, or three-term) form, the approach is often called autotuning. If 
the controller is more complex, for example, when the plant has significant dead time for which 
PID control is not “tight,” a full process model is estimated using system identification methods, 
and an analytic design procedure uses the model to self-tune the coefficients of a fixed control law. 

¢ For time-varying systems: Here the identification algorithm operates all the time to “update” 
the model, and the coefficients of the control law are then automatically adjusted, as shown in 
Figure 33.1. This scheme must be “alert” so as to track variations in the plant’s transfer function. 
This is the full adaptive control approach. 


Figure 33.1 shows the basic structure of a self-tuning controller: the parameter estimator builds up the 
plant model from input/output data {u(t), y(t)}; a control design algorithm takes the estimated model 
parameters 6 and determines the “best” set of controller parameter 9, (such as gains and time constants); 
and a controller applies these gains in a feedback loop. Such self-tuning designs have two time scales: a 
fast inner control loop and a slow outer “updating” loop. 


33.1.1 Examples of Unknown and Time-Varying Systems 


An annealing furnace thermally soaks metal over some prescribed temperature/time profile to attain 
desired material properties. The equation describing the furnace temperature T as a function of heat 
input u is 


dT 
Mcp =u-—6T, 


where m is the mass of the “burden,” which differs from load to load, and £ gives the heat loss from 
Newton’s law of cooling. Hence the time-constant of the system, mcp/B, varies with the burden, so that a 
fixed controller will lead to transients that differ accordingly. For example, a controller tuned for a large 
load would heat up a small mass too quickly. A self-tuned controller that deduces m (or an equivalent 
parameter) during warm-up provides more consistent results. 
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A hydraulically powered shaking table is used to test models of buildings to evaluate their earthquake 
resistance. It is required that the spectral density of the shaking should correspond to that of typical 
earthquakes at the design site. However, the model has mass and other dynamics that affect the behavior 
of the table (just as an inertial load added to a position control affects the loop dynamics). A dynamically 
complex (i.e., more than just PID) self-tuning controller is required to ensure that in closed loop the 
spectral density of the shaking is accurate. A similar problem occurs when testing suspension systems for 
racing cars using hydraulic actuators, where the objective is to replay the surface deviations of a given 
racing track in order to “tune” the suspension for best performance. 

An exothermic chemical reactor generates heat Q = kxe*’, which varies exponentially with temper- 
ature T and depends on the proportion x(t) of reagents left in the mixture. A linear model for small 
excursions around some nominal operating point Ty can be deduced, so that the transfer function 
between u (control) and AT (deviations) is 

1 

~ sc+B— ya’ 

where y = kx exp(aTo). Consider the behavior during a batch with x(0) = 1 initially and where the 
objective is to end up with x = 0 as quickly as possible. First T is small and G(s) is stable. Heat is then 
added via u so the temperature and hence y increase. This might cause the pole of G to become unstable. 
As the reagent “strength” x(t) decreases during the reaction, the plant becomes stable again. In practice, 
many reactors are used for producing a range of chemicals for which variations between batches are highly 
significant. In such cases, a self-tuned controller that adjusts parameters according to the individual batch 
might suffice. Even better performance is possible ifan adaptive controller is used, adjusting its parameters 
as the reaction progresses. 

A materials testing machine has dynamics that depend on the stiffness of the specimen under test. If it 
is testing a manufactured item such as a bump-stop for a car suspension, the stiffness changes radically 
during the cycle. With a fixed controller, the response is either “sluggish” during one phase and acceptable 
in the other, or it can be successively well tuned initially and then oscillatory. The output of a “specimen 
stiffness estimator” can be used to “gain schedule” a PI (proportional-integrator) regulator to get good 
response over both parts of the cycle. This is full adaptive control, as rapid variations in parameters 
are seen. 

The above examples indicate the range of possible controller tuning problems: PI settings for a simple 
fixed plant; a more general controller for a plant with rich dynamics; regulating a slowly changing plant; a 
rapidly time-varying plant for which the dynamics depend on some changing but measurable parameter. 

Successful applications of self-tuning control have been for those cases where engineering knowledge 
leads to a simple model of the underlying dynamics for which bounds on parameter variation can be 
deduced. Early hopes for an effective “general-purpose” self-tuner have not been realized in practice. In 
particular, this is because of the requirement for “persistency of excitation” in the plant’s input/output 
data, which cannot in general be guaranteed. Hence, in the following we concentrate on the fundamental 
ideas; the section “References” contains more details of, for example, multivariable self-tuning designs. 


G(s) 


33.2 Some Simple Methods 


One of the most basic problems in self-tuning is to find a “good” dynamic model ofa plant. Suppose we take 
the classical first-order system G(s) = K/(1+ sT), where K, T are unknown numbers that characterize its 
behavior. The simplest test is to inject a step of amplitude U and inspect the response and its derivative: 


y(t) = KU — e*/7) > He) = etl. (33.1) 


We note two things: (1) y(t) > KU as t > oo, and (2) the tangent at t = 0 has slope KU/T and meets 
the line y= KU when t = T. Hence the “final value” gives K, and the meet of the tangent at the origin 
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FIGURE 33.2 A third-order system with a dominant pole. 
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FIGURE 33.3 Finding K, T, Tq from a step-test. 


with the final value gives T. Now many real plant responses can be approximated by a first-order system 


together with a dead time, giving: 
K 


~ p—Sla 
G(s) Ye last (33.2) 
The response of this model is simply the above first-order response shifted by the dead time Ty, so that 
Tq is found just by inspecting when the output starts to move after the input step U is injected. To give a 
sense of the accuracy of this approximation, Figure 33.2 shows the step response of 1/(s + 1)(1+0.1s)’, 
having a pole at s = —1 and two “fast” poles at s = —10. It is seen that the initial dynamics look closely 
like those of a dead-time of about 0.2 seconds. 
How can we use the above simple idea for finding K,T, Tg? What we can look for is the point of 
inflection of the output curve and extrapolate, as shown in Figure 33.3. 
This is easily done by inspection, but such a procedure is actually quite hard to do in a computer. Even 
more tricky is to derive a method that is reliable with real data where the output response is corrupted 
by noise. We might be able to get “reasonable” values of the model parameters “by eye,” but a computer 
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algorithm will be completely confused. What is necessary is a method that considers the overall response 
without getting locked into local features. Hence, looking for points of inflection is not the answer. Revert 
to the original first-order system, and write it as a differential equation: 

dy 


T— = Ku(t). 
prea u(t) 


Integrating with respect to t from t = f1 to t = 12 gives: 
12 12 
Tly(t2) — y(#1)] +f ydt= x | udt. (33.3) 
al al 


Given a series of output samples y(0), y(h), y(2h),...,y(ih),..., where h is the sample interval, we can 
approximate the integrals in Equation 33.3 by, say, using the rectangular approximation: 


kh i=k-1 
/ yat~h )~ y(ih), 
jh i=j 
i=j 
leading to: 
Ta, + a, = Kb, (33.4) 


where 4}, @, bj are available numerically. By repeating the procedure over another time period, a second 
such equation results, allowing the unknowns K, T to be deduced. We note that the use of the integrated 
equation to a certain extent “smoothes out” the noise. 

Suppose (as with the annealing furnace) it is desired that the closed-loop transfer function is to be 
first-order with unit dc gain and a fixed time constant T, irrespective of the values of K,T. With a PI 
controller of the form C(s) = K,(1 + 1/sTj), the choices T; = T and K, = T;/ (K T,) are appropriate. This 
completes the self-tuning design. 


33.2.1 A Plant with an Unknown Time-Varying Gain 


Consider now controlling an unknown-gain plant using discrete-time methods. At the sample instants, 
the output measurement y(t) is made via an analog-to-digital converter, and the control u(t) is calculated 
and applied via a digital-to-analog converter. We treat the general self-tuning design problem in two 
stages: 


1. The design of a controller assuming known transfer function parameters 
2. The estimation of the plant’s dynamic parameters from the input/output data sequences 


For discrete-time systems we will use a mixed Z-transform/shift-operator notation, where the z is con- 
sidered to be the forward-shift operator: zx(t) + x(t + 1). The current sample of a variable x is x(t), the 
previous sample is x(t — 1), and so on. In particular, A is the backward-difference operator A = 1—z™!, 
giving Ax(t) > x(t) —x(t— 1). 

A plant with known gain K has a sampled-data model y(t + 1) = Ku(t), as the control asserted at 
sample t affects the output measurement one sample later, so given the set point w(t) the “best” open-loop 
control for attaining this value one sample later is clearly u(t) = w(t)/K. But we generally want to have 


closed-loop control so we compute an error signal: 
e(t)=w(t)—y(t) or e(t)=w(t)— Ku(t— 1), 


in the usual way. But the “best” control is such that w(t) = Ku(t), so replace w(t) by Ku(t) and rearrange 
to get the feedback control: 


u(t) =u(t—1)+e(t)/K, or Au(t)=u(t)—u(t—1)=e(t)/K, 
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1@:5 


u(t) = axel (33.5) 


where e(t) is the system error. Hence, the controller has the transfer function 1/K A: an integrator. We can 
modify the controller gain by a factor wt to give /K A for which the “gain” p = 1 gives “ideal” single-step 
response to a change in w. This is the first step in a self-tuning design. 

Now we design an estimator for the unknown plant gain parameter K. The basic idea is that if 
the output = Gain*input, then the Gain estimate = output/input. The way we will do it, however, will 
generalize to larger problems. The idea is to use the model to “predict the present” and to compare the 
prediction with the actual measured plant response. 

A prediction model gives a forecast (t+ 1|t) of the future output y(t+ 1) depending on available 
input/output data and values of the model parameters. In our case, the forecast of the current output 
depends on the existing value of the gain estimate K(t — 1) calculated at the previous sample: 


$(t|t — 1) = K(t— 1)u(t — 1), 
and we define the prediction error €(t) to be 
e(t) = y(t) — p(t|t — 1). (33.6) 


The estimator updates at each t its “best guess” of plant gain. An “open-loop” best estimator would give 
K(t) = y(t)/u(t — 1); but, as with control design, we prefer a closed-loop algorithm of the form: 


K (new) = K(old) + f (data).e(t). 


Hence we write, as for the controller design: 


>», yt) _ [K(t= 1u(t-1) + €)] 
B= u(t—1) u(t — 1) ; 
noting that 
() =H) + (8) = RE — 1) + —* es) (33.7) 
a ae le Sa Nae=D- : : 


where the “estimator gain” ) is 1 for “optimal” performance. The sequence of operations is first to estimate 
e(t) by comparing the old model’s prediction j(t) of the newly acquired output data y(t), then to update 
the model parameters depending on the chosen value of “adaptive gain” i. 

To get the full self-tuner we just couple the estimator of plant gain of Equation 33.7 to the feedback 
controller of Equation 33.5 by passing K from one to the other. Note the resemblance between controller 
and estimator as shown in Figure 33.4: both are integrators. The use of feedback in control is for good 
performance despite “bad” models and disturbances; similarly, with the estimator we can get good results 
despite noise added to y. Just as with control where loop gain 1 is reduced to improve stability, we reduce 
d to improve the estimator’s robustness. With “optimal” values of and 2, the algorithm takes just two 
successive sample times to get perfect control: step 1 gives an accurate K and step 2 attains the set-point. 
As an example, consider a plant that at t = 0 has K = 2 and our model K(0) = 1. At t = 1 the set point w 
is made 4, and at t = 3 the plant gain changes to 1. Iterating the equations for the first five samples gives 
Table 33.1. 
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w(t) e(t) ri u(t) y(t) 
com cele -1 a 
+ KA ed 
= Controller 
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+ u(t—1)A 
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FIGURE 33.4 The controller and estimator as feedback loops. 


33.3 Plant Models 


A mathematical model of a plant is required in order to design a controller. The design procedure needs 
to be analytic so that a unique control comes from a given model and performance criterion. In most 
cases, we consider a model as a predictor of the behavior of the system, e.g., its output response y(t + 7) 
to inputs u(t — j). The model will have a set of parameters 0; identification will provide values for these, 
which can be fed into the control design. It is to be hoped that the model can in fact represent the “real” 
behavior of the plant. (Aside: A distillation column might be described by four hundred simultaneous 
nonlinear differential equations—far too complex for our needs. Fortunately, it is often well represented 
by a second-order transfer function with dead time.) 
We adopt a model structure M(*) that is parameterized by the set 0; for example, we could choose: 


b 
M(6) = ae ie., 0 = {ao, a1, bo}, 


but this would not be sensible as an infinite number of values 6 would give the same transfer function. 
Clearly it is better to divide through by a; and assume: 


b 
M(0)=—"—, ite, 9= {ap, bo}, 
+ ao 


s 


with one fewer parameter to estimate and a unique answer obtainable. 
The choice of model structure is therefore important in the design of general-purpose system identifi- 
cation or self-tuning schemes: 


« Can M represent a general class of plants (e.g., unstable, lightly damped, high order, . . .) so that 
the self-tuner does not need to be redesigned for each case? 


TABLE 33.1 The Signals in the Simple Self-Tuner 


t w y e K K y € u 
0 0 0 0 2 1 0 0 0 
1 4 0 4 2 1 0 0 4 
2 4 8 —-4 2 2 4 4 2 
3 4 4 0 2 2 4 0 2 
4 4 2 2 1 1 4 -2 4 
5 4 4 0 1 1 4 0 4 
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¢ Does M usea minimal set of parameters so that the “true” input/output behavior is given by unique 
values of 6? 

¢ Is the structure of M such that estimation is simple? So that it is robust to bad assumptions (e.g., 
about the number of poles and zeros in the real plant)? 


We shall concentrate on dynamic models in which the underlying plant is continuous time and assumed 
to be in the locally linearized form: 

y(t) = sult Tg) + d(t), (33.8) 
where Tq is the dead time and d(t) is an unmeasured disturbance. The inclusion of a dead time is for 
two reasons: (1) many industrial processes involve mass transport with corresponding delays, but also (2) 
plants with complex dynamics (e.g., many poles) can be approximated by a low-order model with dead 
time. The disturbance reflects reality: it corresponds to any factor that affects the output not included in 
the G(s) such as measurement noise, “unmodeled dynamics,” nonlinear effects, real disturbances (such as 
load torques) acting on the system. It will be found that estimation is a compromise between “alertness” 
to changes in the plant’s G(s) and susceptibility of the estimator to noise (fast estimators adapt rapidly to 
new output data, which sadly could be noise rather than real signals). 

As most identification and self-tuning algorithms are implemented digitally, we shall assume discrete- 
time models obtained from G(s) via sampling. These models can come in many forms, though all are 
usable as predictors. The simplest general-purpose form is the pulse response or weighting sequence. 
Consider injecting a unit pulse (ie., of height 1 and over the sample interval h) into the plant and 
sampling the output to get a sequence {hj}. For a general input u(t), the sampled output y(t) is then given 
by the convolution sum: 


[o,0) 
y(t) = So hyu(t - i). (33.9) 
i=l 
The only assumptions here are superposition and open-loop stability: the arbitrary linear plant has a 
parameter set 6 = {h;} being the sampled points on the pulse response. 

Theoretically there are an infinite number of parameters in this model, but in practice we truncate after 
some point N, assuming [h; = 0,1 > N] and where hy is zero, as the plant cannot respond instantaneously. 
If there is a dead time of k samples between a control and its initial effect on the output, then [hj = 
0, fori=1...k] also, so that the model handles dead time simply by having leading coefficients of zero. 

One problem with FIR models is that they require a very large number of parameters accurately to 
represent “stiff? dynamic systems (i.e., with fast and slow modes in the same plant), or even a simple 
lightly damped pole pair. The sample interval h must be less than the smallest time constant of interest, 
and the model “length” must be such that Nh exceeds the plant settling time, which is typically five times 
the largest time constant. Hence, even with only a 1:10 range of time constants, at least 50 parameters 
may be necessary. Nevertheless, the associated computations are very simple (easily embedded in VLSD, 
and FIRs are commonly used in signal processing and some process control designs. It sometimes proves 
to be useful to consider the input to a plant as a series of increments Au(t) = u(t) — u(t — 1) (i.e., steps or 
control “moves” as in a stepper-motor) and the response (by superposition) to be 


y(t) = s,Au(t —1)+s,Au(t—2)+---+s;Au(t—i)+---, 


where s; is the ith point on the plant’s unit-step response. It is easy to show that the predictor that gives 


the next plant output is 
N 


y(t +1) = y(t) +s, Au(t) + > hyAu(t i). (33.10) 
i=l 
The predicted output y(t + 1) is the sum of three components: the current output y(t), the forced response 
due to the current control “move” Au(t), plus the free response due to previous control moves. 
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33.3.1 Transfer-Function or Difference-Equation Models 


By far the most popular model in self-tuning control is the DARMA (deterministic autoregressive and 
moving average) or general difference-equation form: 


y(t) + ary(t — 1) + aay(t — 2) +--+ + anay(t — na) = byu(t — 1) 
+ bou(t — 2) +-+++ dypu(t — nb). (33.11) 
Defining A(z!) and B(z~!) to be polynomials in the backward-shift operator: 
A(z7!) = 1+ ayz7} +--+ + aygz—™ (33.12) 
Biz!) = bz +--+ byez ™, (33.13) 


this can be written 
A(z~!)y(t) = B(z!)u(t), 


ie., a transfer function 


(33.14) 


The values of the parameters are obtained by taking Z-transforms of G(s) (+ZOH) as previously shown. 
Note that: 


¢ There is a unique correspondence [via the mapping z = exp(sh)] between the continuous- and 
discrete-time poles; the degree na is the same as that of G(s). A pole s = —a in the left half-plane 
(LHP) in s (stable) maps into z = exp(—ah); ie., within the unit circle. 

¢ There is no simple mapping of zeros. Indeed, even if G(s) has all its zeros in the stability region (LHP), 
this does not mean that G(z~') will be likewise (in unit circle). It is, in fact, extremely common for 
so-called nonminimum-phase discrete-time models to appear (zeros outside unit circle), e.g., when 
there is fractional dead time or when controlling a high-order plant with a small sample interval h. 


33.3.2 Incorporating Disturbances 


In continuous-time random processes it is useful to define “white noise”—a signal with a constant spectral 
power at all frequencies. In discrete time, the corresponding signal is a sequence of random independent 
(uncorrelated) variables with zero mean and common variance o? [hence called a (0,07) uncorrelated 
random sequence (URS)]. Here we do not have an “infinite variance” signal, but instead something easily 
produced by a random signal generator such as RAND in MATLAB®. Such a signal has the following 
properties: 

Ee(t)=0; Ee(t)= 02; Eeli)e(j)=0, fori Fj, (33.15) 
where € denotes the expectation operation. As in continuous time, a general stationary discrete-time 
random process is modeled by white noise passing through a transfer function in z. 

Suppose the controlled part of a plant is G = B, /A, and the output y is affected by additive disturbances 
C, /Dye(t). Then we have: 


By Ci 
t)= —ul(t —e(t), 
y(t) a + pe ) 
which, when multiplied up, gives: 
A,Diy(t) = B,D,u(t)+ C, Aje(t). 
Hence, we make the common overall plant assumption of the CARMA (controlled autoregressive and 
moving average) model: 
A(z~!)y(t) = B(z)u(t) + C(z~)e(t), (33.16) 


where e(t) is a URS sequence of independent (0,07) random variables. The corresponding difference 
equation is then generated using the polynomials A, B, C. 
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However, the CARMA form is insufficient to characterize offsets for which u = 0 is not accompanied 
by y =0. A model could include deviations from the mean levels uo, yo, say, but in control loops these 
mean levels would not, in general, be constant or known. Indeed, it is often found that disturbances are 
“steps”; for example, when passengers enter an elevator, there are steps in the load torque acting on the 
motor: these correspond to shifts in mean levels. One way to deal with this problem is to add an offset 
parameter d to the output. In a full model used in self-tuning, the value of d would need to be estimated 
along with the other (dynamic) parameters in the A, B, C polynomials. 

In practice, the disturbance is likely to be a combination of factors such that no consistent values of the 
parameters C(z~!) can be estimated. But it is known from the internal model principle that disturbance 
elimination is best achieved by having an internal model of the disturbance in the control law. Inspection 
provides the following “theorem”: the ubiquitous nature of PID regulators in industry implies that an 
integrator is the internal model of most practical disturbances. Hence, the best general assumption is of a 
CARIMA (integrated) model in which random disturbances are integrated: 


A(z~!)y(t) = B(z)u(t) + C(z~!) x(t), (33.17) 


where x(t) is of the form a/A for deterministic step disturbances, or Ax(t) = e(t) [with e(t) a URS] for 
random disturbances. Incorporating into the model gives an incremental form: 


A(z~!)Ay(t) = B(z~!)Au(t) + C(z~)e(#), (33.18) 


or 


y(t) = y(t — 1) — ay Ay(t — 1) —-- + dng Ay(t — na) + b) Au(t — 1) +--+ byp Au(t — nb) 
+ e(t)+cye(t—1)+-+++cnce(t — nc). (33.19) 


Note that the model deals with increments of the input/output data such as Ay(t — 1) = y(t — 1) — y(t — 2) 
and hence no dc term is involved as A.constant = 0. It is found that injecting a pulse into x (or e) gives a 
step change in y, and injecting a “white” random sequence or URS gives “Brownian motion” for which y 
“drifts” (Brownian motion is quite a good model of stock exchange prices). 


33.4 Recursive Prediction Error (RPE) Estimators 


The job of an estimator is to provide values for the model parameters based on fitting the model’s 
responses to the measured plant input/output data. A recursive estimator updates the estimates at each 
sample instant based on the newly available information. One important point about recursive estimators 
is that the computational load must not increase with time. Note that potentially the amount of data is 
always increasing with more available from each sample. Prediction error methods consider the model 
to be a forecaster )(t|t — 1) of the actual outcome y(t), the difference or residual € = y — j being used to 
correct the estimates. Now define the following n vectors: 


1. 0=[61,...,6,]' are the n unknown plant parameters. 
2. O(t) = [61 (2),...,9,(t)]’ are the corresponding estimates at time t. 
3. x(t) =[x1(t),...,xn()]' are known data associated with the parameters. 


A prediction model generates a forecast j(t) depending on 6(t — 1) and x(t): 


H(t) == H(t|t — 1) = f(t — 1), x(£)). (33.20) 


The models we will consider are all in linear-in-the-parameters (LITP) form with: 


H(t) =D Oi(t — 1) xi(t), (33.21) 


i=1 
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FIGURE 33.5 General RPE structure. 


and where the measured plant output is assumed to satisfy the LITP equation: 
n 
y(t) = >> Oixi(t) + e(t), (33.22) 
i=l 


with e(t) corresponding to “noise” or “disturbances” and assumed to be independent of the data x(t). The 
scalar prediction error is defined to be 


e(t) = y(t) — H(t) = y(t) — f(t — 1), x(6)), (33.23) 
where y(t) is the new output measurement. The general structure is shown in Figure 33.5. 
Typical models that come under the general description of Equation 33.21 are: 
Pulse response 
y(t) = )> hju(t - i) 
i=l 
where: 


@=[hy,ho,...,h,] and x(t) =[u(t—1), u(t —2),...,u(t—n)J’. 


In this model, the parameter vector are points on the pulse response, and the data vector contains n past 
values of the control signal. 


DARMA 
y(t) = —ayy(t — 1) —- +» +b u(t — 1) +--+ + bypult — nb) 
where: 
= [ay,. : Ana, 0}, . : -> Dypl! 
and: 


x(t) =[—y(t—1),...,-y(t— na), u(t—1),..., u(t —nb)]’. 


In this model, the parameters are the difference-equation constants, and the data vector contains past 
values of inputs and outputs. 
Note that the LITP equations can be written concisely as: 


p(t) =x’ (t)0(t — 1), 


and equivalently 
y(t) =x'(1)0 + e(t), 
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so that if the parameter error vector 0(t) is defined to be 6 — 6(t), then: 
e(t) = y(t) — p(t) = x'(t)0(t — 1) + e(t). (33.24) 


This important equation shows that the prediction error, has two components: the model error and the 
unknown disturbance. Our use of this result depends on whether we want to identify a time-invariant 
system (in which case we average over a lot of data to eliminate the effects of noise) or whether we want to 
track a time-varying plant (which is possible only if the signal to noise ratio is large and the variations are 
not too fast). We can associate with the prediction error a cost function J such as J(t) = 0.5e?(t). If there 
is no noise, this cost depends only on 6 and so we can imagine a set of equal-cost contours in the n-space 
of the parameters 6 with the minimum value at the “true” parameters. If there is noise, these contours will 
correspond to average or expected values. 
An RPE algorithm updates the estimates using: 


6(t) = 6(t — 1) +. a(t)M(t)x(t)€(t), (33.25) 
where: 


« a(t) isa scalar “gain factor,” giving the step length. 

¢ x(f)e(t) is along the gradient vector —V/J of the cost-function surface, pointing down the slope of 
local steepest descent. 

¢ M(t) is a rotation matrix that modifies the parameter update direction away from the steepest 
descent route. 


We can reduce the parameter error using a “large” value of the gain a(t), but this increases the effect of 
the disturbance e(t). The compromise between rapid model error reduction and insensitivity to noise 
is a fundamental design issue: in practice, we want rapid initial convergence, followed by good noise 
immunity for steady-state tracking. In view of its long history of success in many applications, the most 
common estimator used in self-tuning is based on variants of the least-squares (LS) method, for which 
the current estimate 6(t) minimizes the “loss function”: 


t 
I(t) = (6(t) — 0(0))'S(0) (t) — 16(0)) + Y° GH) — x (DBC)? (33.26) 


i=l 
It can be shown (e.g., Isermann et al. [8]) that J(f) is minimized by the recursions: 


S(t) = S(t — 1) + x(t)x’(£) (33.27) 
6(t) = 6(t — 1) + S(t) !x(t)€(t), (33.28) 


with 0(0) being the initial “guess” of the parameters and $(0) an assertion about their likely accuracy (ie., 
small entries in S implies low accuracy). It can be shown that: 


S(t)6(t) = $(0)6(0) + [S(t) — $(0)]9, (33.29) 


so that as S(t) increases, the effect of the initial assumptions declines, and a solution is possible only if S 
has full rank: a “persistency of excitation” condition. 
In practice, we prefer to propagate P(t) = $~!(t) to avoid the inversion in Equation 33.28; using P, the 
important recursive least-squares (RLS) equations become: 
Kalman gain vector: 
P(t — 1)x(t) 
~ 1+-x/(f)P(t — 1)x(t) 


k(t) (33.30) 


parameter update: 
Q(t) = O(t — 1) + k(£)e(t) (33.31) 
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covariance update: 


P(t) = [I—k(#)x’(£)]P(t — 1) (33.32) 


The algorithm needs to be initialized. Note that P is proportional to the “errors in the parameters,” 
as P= $~!, so that at t = 0 where nothing (or little) is known, the appropriate choice is yI where y is 
large. It can then be shown that the estimates obtained after n samples using RLS are the same as with 
“off-line” LS. For smaller y, though, slower parameter movement is seen, for that choice indicates the 
chosen 6(0) are fairly accurate. In this way it is possible to start recursive estimation with a reasonable 
“guessed” model rather than simply (0,0,0...)’. 

What happens for large t? Assuming that there are good persistently exciting data, then $ = “4 xx’ 
increases all the time and hence ||k|| — 0. This means that asymptotically RLS loses its “alertness” (though 
by that time there should be convergence to the “true” values despite noise). This is a natural consequence 
of the built-in assumptions: a fixed parameter plant where accurate estimates are wanted. Hence, the 
method is possibly suitable for self-tuning but not for adaptive control. Figure 33.6 shows RLS in action 
with data from a FIR model in which the parameters {h1(t), h2(t)} each change by square waves. The 
input is white noise in “bursts” so it is not exciting during quiescent periods. The estimates do not move 
at all when there is zero excitation and change at an increasingly slow rate even when the data are “rich.” 
Hence, RLS concentrates on accuracy (assuming fixed plant parameters) and not on tracking parameter 
changes. 


33.4.1 Forgetting Factors 


The cost function in LS weights all residuals «(t — j) equally, no matter how far back in the past the data 
were acquired. If it is expected that the model dynamics varies with time, then recent data are more 
significant than old data. Thus, the idea is to “forget” so that the effect of data on the estimates decays 
in time. It is convenient to consider a simple single-parameter continuous-time estimator to explore the 
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FIGURE 33.6 RLS with no forgetting. 
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ideas. As a preliminary, consider the integral: 


t 
T= i e NFO) dr. (33.33) 
0 
Then, from first principles: 
dl 
— =f(t)-al. 33.34 
Prem A a (33.34) 


A fixed forgetting factor exponentially weights past data at a rate a, so that using the continuous-time 
exponentially weighted version of Equation 33.26, the estimate 0(t) minimizes: 


t 
](t) = e~“'s(0)(6(t) — 6(0))? + / e F(X) — B(t)x(A))? dr, (33.35) 
0 


giving, by setting the partial differential with respect to 6 to zero: 
t 


s(t)0(t) = e~%*s(0)6(0) + i. e MEN )x() dd (33.36) 
0 


where: : 
s(t) =e~“'s(0) + i eo MEN) 20) dd. (33.37) 
0 


Differentiating with respect to tf and using the preliminary result gives the recursive equations of 
continuous-time RLS for a single parameter: 


d 

+as=x? (33.38) 
dd x? xy 
i pe PO 33.39 
dt s s ( ) 


with initial conditions {s(0), 6(0)}. The Equation 33.39 shows that, for constant excitation x = X, s(t) 
rises exponentially to X*/a; for zero “forgetting” « = 0, s(t) > oo if the data are persistently exciting. In 
practice, the convention is to use the current prediction error € = y — 9x, so substituting we get as the 
update equation: 

= =6, 33.40 

dt os ( 
It is instructive to find how the modeling error 6 = 0 — 6 changes with time. Substituting for y using 
y = 0x + e and assuming that 0 varies in time we get: 


The left-hand side tries to drive 6 to zero at a rate depending on x”/s while the right-hand drives it away 
from zero. In RLS it is conventional to use the inverse p = 1/s in the calculations. Substituting in the 
update for s we get: 

dp 2,2 dp 2 

——-ap=—px*, or —-+(x*p—a)p=0. 33.41 

Same mia 3 qt POOP (33.41) 
The behavior of forgetting factors such as a is clearly shown in this formulation: the equation has an 
unstable mode when xp < a. If there is no excitation [x(t) = 0], p(t) increases exponentially at a rate 
determined by a. If x is constant p increases until p = a/x? and then becomes constant. Hence, clearly 
one method to avoid “blow up” is to add a small quantity x5 to x? in the updating equation for p. 
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33.4.2 Variable Forgetting Factors 
The loss function J can be written in terms of €: 
t 
I(t) = Jo(t) + i, eM MEY dd. 
0 
Suppose now that estimation is perfect (i.e., 6 = 6) for all t so that €(t) = e(t) always and the variance of 
the noise is of. Then the expected value of the irreducible or “ideal loss” is given by: 


t 2 
E({Jopt} = / eth) 2 d= 0 —exp™). 
0 a 


For large t, this approaches the value of /ca = T-0, say, where T, corresponds to the “asymptotic averaging 
time”; i.e., the period of past time that contains the data “most influential” in providing the estimate. 
Consider the differential equation for J, which is: 


d 2 

—+ajJ=e€*(t). 

it J (t) 
We would expect the average value for € tobe greater than Tas given that €(t) = x0(t) + e(t). In particular, 
we expect that €” >> of when the estimate is far from the true value of 6. One way to proceed is to assert 
that the value J (or 1/J) contains the “information” about the parameter and that this should be constant 
in the steady state. This implies that a good value for « is obtained by making oJ = €? = Wopr, giving: 


1 () 
Se [Pay 
T- 00 
and the update equation for s now reads: 


ds. T-fe\? 5 
=x: 33.42 
dt - T- ( 00 ) ae ( ) 


Hence, adaptation is faster with a prediction error that is large compared with the assumed SD of the 
underlying noise. 


33.4.3 Forgetting with Multiparameter Models 


For discrete-time estimation we write B = exp(—ah), where h is the sample interval (small a > B ~ 
1 —ah,) and then the RLS equations become: 


P(t — 1)x 


6(t) = 6(t — 1) + k(t)e(t) (33.44) 
P(t) = [I—k(t)x’(t)|P(t — 1)/8 (33.45) 


A useful measure of the amount of data effectively contributing to the current estimate is the asymptotic 
sample length (ASL) or time constant, given by 1/(1 — B). A rapidly varying system might need an ASL of 
20 samples (8 = 0.95) to more than 1000 (8 = 0.999). Figure 33.7 repeats the experiment of Figure 33.6 
but using B = 0.95. Again, when the input is not exciting, the parameters freeze (and P “blows up”), but 
otherwise track changes rapidly. Moral: Use forgetting, but then make sure the input perturbs the plant. 
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FIGURE 33.7 RLS with forgetting. 


33.5 Predictive Models 


Industrial processes have characteristics that make advanced control useful: 


Disturbances: Fluctuations in the raw materials and the operating environment; sensor noise. 
Dead-time: The effect of the current control is not seen in the measured response for a number of 
samples, because of material transport times. 


Typical examples of processes like this are: 


Steel rolling: Controls are via screws on the stands; gauge responses are after the end of the mill; x-ray 
gauge measurements are noisy. 

Papermaking: The controls are on the head box at entry to the Fourdrinier wire; the basis weight of 
the dried paper is measured after the heating rolls. 

Strip annealing: Inductive heating causes temperature changes, later measured by a pyrometer with 
large random fluctuations in its signal. 

Distillation: The multiple lags (several hundred) arising from the thermal behavior of each tray appear 
like dead time; composition is measured by a chromatograph with sampling delays; ambient thermal 
variations induce disturbances. 


Delays cause phase lag, which means that PID control gains must be reduced. Predictive control can 
give “perfect” control provided the delay is known; even better performance is obtainable if the distur- 
bance process can be predicted also. Consider the problem of predicting the plant output with its two 
components: 


Deterministic effects: Old inputs and outputs give initial conditions, from which the “free response” 
can be determined via the model. The “forced response” is the additional effect due to current and 
future controls. 

Disturbances: Old URS values e(t — i) can be reconstructed from the model and known data. The free 
response can then be computed; as nothing is known about future white noise, the best approach is 
simply to assume its mean value of 0. 
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Consider an autoregressive (AR) process driven by white noise giving the measured output y(t): 


y(t) =0.9y(t—1)+e(t), or y(th= e(t). 


— ee 
(1—0.9z71) 
By expanding as a power series, the model can be written in pulse form: 
y(t) = e(t) + 0.9e(t — 1) +0.97e(t — 1) + 0.9% e(t — 2) + ---. 
Consider its value two steps into the future: 
y(t +2) = e(t +2) + 0.9e(t + 1) + 0.97e(t) + 0.9% e(f —1)+---. 
As e(t + 1), e(t + 2) are not known at time f, the best prediction of y(t + 2) is 


0.97 


>(t + 2\t) = 0.9%e(t) + 0.9% e(t — 1) +--- = —— 
Ht + 2|t) = 0.9% e(t) + 0.9 e(t — 1) + 055 


e(t). 


But the noise model gives e(t) = (1 — 0.9z~!) y(t), giving the simple result that }(¢ + 2|t) = 0.97 y(t) 
(the free response of the noise model). The error in the prediction is 


p(t + 2|t) = e(t +2) + 0.9e(t + 1). 


To find the variance or mean square of this signal, simply square and take averages, taking Ee(t + i)? = 
07; Ee(t + ie(t +j) = 0, and the following should be noted: 


1. The prediction error (p.e.) is independent of the “known” data [in this case, e(t), e(t — 1), .. .]; i-e., 
the maximum possible information has been extracted. 
2. The variance of the p.e. is (1 + 0.97)o, where o? is the variance of the URS e. 
The p.e. variance increases with the prediction horizon; here, two-steps ahead. 
. “Sluggish” disturbances (pole near 1) are predicted with more accuracy than rapidly moving 
disturbances. 


mY 


To generalize this example, consider a noise model in moving-average (MA) form: 
y(t) = N(z7")e(t), 
so that: 


y(t +k) = (Lt mz oe tgp elt +k) + (npez* + tgpiz 1 ++ elt +k) 
=Ng(z"!)e(t +k) + Ni (z")e(t). (33.46) 


The disturbance is split into “future” and “past” components, and the prediction uses known data: 


(t+ kt) = Ny(z)e(t) = nge(t) + npyye(t-—1)+---. 
How does this procedure work with a transfer function (ARMA) structure: 


Ce 
~ A(z!) 


y(t) e(t)? 
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Performing long division by A and stopping after k terms gives: 


In fact, instead of doing long division we multiply each side by A to get: 


C(z7}) = E(z)A(z7!) +2 F F274), (33.47) 


This key equation is known as a Diophantine (or Bezoutian) identity from which E and F can be obtained 
for given A, C, k by equating powers of z~!. For example, consider: 


(1—0.927!)y(t) = (1+ 0.7z7')e(t) > A=(1—0.9271); C=(1+0.7z7). 
Hence, the Diophantine identity of Equation 33.47 for k = 2 becomes: 
(1+0.727!) = E(z71). — 0.9271) 4+. 27-7 F(z7}), 


or 
(1+0.7z7!) = (eg teyz!)(1 — 0.927!) +2 7fy. (33.48) 


Equating coefficients of increasing powers of z~! in Equation 33.48: 
Oe ee pnts bys = ‘i 
z:l=e3z :0.7=-0.9+e; 


z?*:0=—0.9e +f. 


Hence we have: 
eo = 1: y= 1.6; fo = 1.44, 


1.e., 
E(z7!) =14+1.6z7! and F(z7!) = 1.44, (33.49) 


and so the two-step-ahead prediction becomes: 


F 
H(t + 2|t) = Nez elt) = Felt) = (t). 


- e 
(1— 0.927!) 


But we can reconstruct e(t) from the measured value of y(t) and the inverted model: 


A(z7!) 
t= t), 
e(t) Cen)" ) 
giving the predictor: 
p(t+2|t) = pat es (t) (33.50) 
y ~ 1407208" ; 


The prediction error j is given by E(z~!)e(t + 2), or: 
H(t +2|t) = e(t +2) + 1.6e(t + 1), 


which has a variance of (1 + 1.67)o” = 3.5607. It is interesting to note that the actual variance of y(t + 2) 
is 14.4707, meaning that our predictor “explains” roughly three fourths of the output variance. 
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33.6 Minimum-Variance (MV) Control 


A growing requirement in manufacturing is guaranteed and quantified quality, as measured, for example, 
by the proportion of a product lying outside some prespecified limit. In continuous processes, such as 
papermaking, it is important that the output (at worst) exceeds some lower quality limit (e.g., thickness 
of paper). To ensure that this is so, the average thickness must be set greater than the minimum by an 
amount dependent on the variance of the controlled output. Hence, if this variance is minimized, the 
manufacturer can reduce the average, as shown in Figure 33.8. 

[Aside: the worst manufacturers sometimes make the best product. If they have a large spread in quality, 
they have to test everything and reject out-of-spec items; i.e., those below the lower statistical limit (LSL) 
of the figure. Hence, the majority of sales is at a higher quality than really needed. It is best to be “just 
good enough” = low spread of quality, and hence be profitable. ] 

Consider the plant with dead time k samples and with model: 


A(z~!)y(t) = Bz )u(t — kb) + C(z )e(t), (33.51) 
or, dividing: 
Biz"! Ci 
y(t+k) = rau ! Spel +h. (33.52) 


The second right-hand side term is the effect of the disturbances on the output, which can be predicted 
using the ideas of the previous section; the first term is the effect of the control (which by assumption can 
affect the output only after k samples). The idea of MV control, in essence, is to choose the control u(t) 
that will counteract the predicted disturbance at time t + k. The development first solves the Diophantine 
identity Equation 33.47 to provide E(z~') and multiplies: 


A(z~!)y(t +k) = B(z")u(t) + Cz e(t +) 


(each side) by E(z~!) to give: 


EAy(t + k) = EBu(t) + ECe(t +k) 


or, as EA = C—z7*F from Equation 33.47: 


Cy(t + k) — Fy(t) = Gu(t) + ECe(t +k), 


By) Required PO) 


average 


Smaller variance 
hence a smaller average 
for same off-spec 


% below 
<— 


ie 


‘\. LSL y LSL y 


FIGURE 33.8 Using minimum-variance control. 
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where G(z~!) = E(z~!)B(z7!). Hence, we have the equation: 


Fy(t) + Gu(t) 


y(t+k)= + Ee(t +k). (33.53) 


The first term on the right-hand side is the k-step-ahead predictor and the second is the prediction error. 
Hence, we can write the key prediction equations: 


k step prediction: 
aoe F(z")y(t) + Gz") 
Wt+ kit) = Ce u(t) (33.54) 
k step error: 
H(t +k|t) = E(z')e(t +k). (33.55) 


As an example take the plant: 
(1 — 0.927!) y(t) = 0.5u(t — 2) + (1 + 0.7z7!)e(t), 
where we have already solved the Diophantine identity of Equation 33.47 giving Equation 33.49 and so: 


1.44y(t) + (0.5 + 0.8z~!)u(t) 
(1+0.7z7!) 


yt+2)= + (1+ 1.6z7!)e(t+ 2). 


The MV control is then easy to determine: Simply choose u(t) so that the first right-hand side term 
becomes 0; all that remains on the controlled output is the prediction error, which cannot be minimized 
further as it is comprised of only future white noise components e(f +1) and e(t+2). The feedback 
control law for our example is then: 


1.44y(t) + (0.54 0.82~!)u(t) = 0, 


or: 
u(t) = —1.6u(t — 1) — 2.88y(t). 


In the general case, the control becomes: 


F(z7!) 
z)E(z7!) 


F 
u(t) = gi) =~ 3 y(t); (33.56) 


this controller cancels the zeros of the plant transfer function. In closed loop, the characteristic equation is 
FE 2k -k 
1+—z “BA=0, or B(EA+z “F)=0, 
BE 
so that, using the Diophantine identity of Equation 33.47, this reduces to: 
B(z!)C(z7!) = 0. (33.57) 
The closed-loop modes are defined by those of C (which, in fact, are stable) and of B(z~!). There 
is, therefore, a potential instability problem with MV control in cases where B has roots outside the 
unit-circle stability region (so-called nonminimum-phase zeros) as these appear as unstable poles of the 


closed loop; such nonminimum-phase zeros occur much more frequently in discrete systems than in 
continuous-time control. 
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33.7 Minimum-Variance Self-Tuning 


All the machinery is now available for self-tuning: connect a parameter estimator to an MV controller by 
solving the Diophantine identity of Equation 33.47. Note that MV controller design requires knowledge 
of k, A, B, and C. In difference equation terms the CARMA plant model is 


y(t) = —ayy(t —1) —---+ bou(t —k) +---+e(t) +ce(t-—1)+---, 


but the standard estimators can estimate only A, B; the driving noise e(t) is not measurable and, hence, 
cannot be placed into the x-vector to estimate C. There are methods (such as extended least squares) for 
estimating C, but these tend to be unreliable in practice. However, it transpires we can obtain self-tuned 
MV (giving a self-tuning regulator) by using a standard LS estimator without needing knowledge of C (in 
effect, assuming C = 1)! 

There is potentially a further problem: the effect of feedback control on the parameter estimates. 
Suppose, for example, that a plant: 


y(t) = ay(t — 1) + bu(t — 1) + e(t) (33.58) 
has a simple proportional controller (with zero set point): 
u(t)=—ay(t), or ay(t—1)+u(t—1)=0. (33.59) 


Then adding a fraction t of Equation 33.59 to Equation 33.58 gives: 


y(t) = (a+ pa)y(t — 1) + (b+ p)u(t — 1) + e(t). 
If we now use an estimator based on the two-parameter model: 
y(t) = 8 y(t — 1) + O.u(t — 1) +e(t) =x'0+e, 


then 6; =(a+a), 6, = (b+) will be obtained, where t is arbitrary. Hence, if we use LS estimation 
in a closed-loop mode, the estimated 6 does not converge to a unique point but to a line where the 
estimated parameters can wander up and down in unison. This is a problem of using closed-loop data 
with only internal signals such as e(t) stimulating the loop; to get an consistent estimate we must do one 
of the following: 


1. Use externally generated test signals, such as step changes in set point. 
2. Have a controller that is more complex (higher order) than the plant. 
3. Have a time-varying controller. 


This third solution is appropriate for self-tuning, though it is still best to make the data “rich” by 
exciting the plant with external signals (e.g., a PRBS added to the set point). 

How is it that we can use LS? The key idea is not to go estimate — design — controller (giving what is 
called the indirect approach), but instead to proceed estimate — controller (the direct approach). What 
are estimated are the controller (in fact, the k-step-ahead predictor) rather than the plant parameters. 
How this is done is seen below. 

Recall the prediction Equation 33.54 when multiplied up by C(z~?): 


(ltcjz 7} +e2-* +--+ p(t +k|t) = Fy(t) + Gu(t), 


giving at time t: 


H(t|t — k) = Fy(t —k) + Gu(t —k) — > ciz‘P(t|t — k). 


33-22, Control System Advanced Methods 


But the point about MV control is that it makes the prediction zero by correct choice of u. Hence all 
the terms in the sum on the right-hand side are set to zero by previous controls, so that from: 


y(t) = p(tlt — k) + p(t|t — k), (33.60) 


we have: 


y(t) = F(z7!)y(t —k) + G(z7!)u(t — k) + E(z™")Je(t). (33.61) 


This is the crucial equation: it obeys the LS rules of an LITP model with: 


x(t) =[y(t—k), y(t —k-—V),...,u(t—k), u(t —k—1),...] (33.62) 
9=[fo, fir---20 g---], (33.63) 
and, most importantly, the data x(t) are independent of the error term as the data are from t — k back- 


wards, whereas E(z~!)e(t) finishes at ex_;e(t — k + 1). Hence, LS leads directly to the required FandG 
parameters, so we get a self-tuner with feedback law: 


F(z7)y(t) + G(z7!)u(t) = 0. (33.64) 


The procedure, then, is as follows: 


1. Assemble old data into the x-vector as in Equation 33.63. 
2. Use RLS to get 0 = F,G. 
3. Use the estimated parameters in the feedback law of Equation 33.64. 


Of course, the above is simply a plausibility argument; in fact, the algorithm can by lengthy algebra be 
shown to converge to give the required control signals; i.e., satisfying the self-tuning property. The speed 
of convergence is found to depend on the roots of C(z~!). As an example, consider the first-order system: 

(1 — 0.927!) y(t) = 0.2u(t — 2) + (1 +. 0.927" )e(t), 
which has the two-step-ahead prediction equation: 
(1+ 0.927!) p(t + 2|t) = 1.62y(t) + 0.2(1 + 1.827! )u(t), 
for which the MV controller with known parameters is 
1.62y(t) + 0.2u(t) + 0.36u(t — 1) = 0. 
The corresponding model to estimate in self-tuning is 
y(t) = foy(t — 2) + gou(t — 2) + giu(t — 3) + €(t). 


The estimator for a self-tuner will have data and parameter vectors: 


x = [y(t — 2), u(t — 2), u(t — 3)]' 
6 =[fo,g0,gi]) > [1.62, 0.2, 0.36]. 
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The system was simulated for 1000 samples, the first 500 being “open loop.” At t = 500 the self-tuner was 
switched on, giving the results seen in Figure 33.9. Observe how the variance has been reduced by STMV 
and how the estimated parameters “wander about.” 

As discussed above, the estimates of the parameters are not unique, as the control Equation 33.64 can 
be multiplied by an arbitrary factor 1 without affecting u(t). In principle, this is not a problem, but to 
avoid excessively large or small estimates we can “fix a parameter” to a guessed value and estimate the 
others. Typically, the fixed parameter is the value of go: the multiplier of the current control u(t), whose 
nominal value is bo. Suppose go is the choice. In our example, it means that the model becomes: 


y(t) — gou(t — 2) = y(t) = foy(t — 2) + giu(t — 3) + €(¢). 


Then the model to use in RLS has data and parameter vectors: 


x(t) = [y(t —k), y(t-k-1),...,u(t-—k—-1),...] (33.65) 
= [y(t — 2), u(t — 3)]’ (33.66) 

in our example; 
6=[fofir--.8-- Vs (33.67) 


and with “output” y(t) = y(t) — gou(t — 2). The control to use is like Equation 33.64 but is based on the 
chosen fixed gp and the remaining estimates: 


u(t) = LE) fiy(t—i) +) Giult — 11/20. 


Minimum variance ST: estimated bp: plant output Control input 
20 T 40 T 
10+ 20 4 
‘I 
~10+ Y 4 20 
—20 : -40 : 
0 500 1000 0 500 1000 
SD 1st half: 4.001 2nd half: 2.15 a: 0.9 b: 0.2 c: 0.9 k: 2 
x Tuned controller parameters 
T 
2 Sa 
0 
25k 
—-4 
0 500 1000 


Forgetting factor: 0.98 


FIGURE 33.9 MV self-tuning, all parameters estimated. 
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A self-tuner based on this idea is coded in MATLAB as: 


% M-file for the simple first-order A-W 


oe 


minimum-variance self-tuner. 


nt = 1000; na = 1; 
th=zeros(nt,np); P = 100*eye(np) ; 
II = eye(np); 


nb = k-1; np = na+nb; 


for i=k+1:nt, 


e(i) = rand; 
y(i) = a*y(i-1) + b*u(i-k) + e(i) + c*e(i-1); 3 plant 

if i> nt/2, 

x = [y(i-k) u(i-2*k+1:i-k-1)]; % data vector 

ep = y(i)-gO*u(i-k) - x*th(i-1,:)'; % pred error 

kk = x*P/ (beta+x*P*x’); % RLS 

th(i,:) = th(i-1,:) + kk*ep; % 

P=(II - kk’*x) *P/beta; % update 

u(i) = - (th(i,1)*y(i) + u(i-nb:i-1)*th(i,2:np)’)/g0; 

end; 
end; 


eg ate oP (33.68) 


This means that a “large” value of the fixed parameter is safe. 

Figure 33.10 and Figure 33.11 show the behavior of a self-tuner with the parameter go set to 0.4 and 
then 2. Both are perfectly well behaved, though with slower convergence for go = 2. Note that in the 
original example with no “fixing” there is a point where the parameter estimates “jump.” It is found that 
without fixing and with no external signals a self-tuner tends to “burst” like this every now and then. 

Generalized minimum-variance (GMV) control was developed to overcome the problems of (1) MV’s 
instability when B is nonminimum-phase and (2) the large control variance produced by MV, particularly 
when using “fast sampling.” An auxiliary output o(t) is defined: 


(t) = P(z")y(t) — R(z')w(t — k) + Q(z" )u(t —k), (33.69) 
where P, Q, R are design polynomials whose choice gives a range of possible closed-loop objectives; see 


Harris and Billings [7] or Wellstead and Zarrop [11] for more details. GMV self-tuning simply uses the 
same approach as developed for MV: 


1. Estimate polynomials F, G, H in the predictor model: 
b(t) = F(z7!)y(t — k) + G(z u(t — k) + A(z!) w(t — k) + (8). (33.70) 
2. Using the estimates from Equation 33.70, compute the control: 


G(z“ u(t) = -[F(z) y(t) + A(z) w(e)]. (33.71) 
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FIGURE 33.10 Simulation of a simple self-tuner with a fixed parameter go. 
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FIGURE 33.11 Simulation of STMV with a larger fixed parameter. 
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w(t) =0 F(z) u(t) _~ B(z7}) y(t) 
r Ge) | 2 Ae) 4 


FIGURE 33.12 Feedback controller for pole placement. 


A simple case of GMV is when P = R = 1 and Q = 3 for which it can be shown that the cost: 
iN 
Joy = El(y(t +) — w(t)? + OIE] (33.72) 
0 


is minimized, where the expectation is conditional on data available up to time t. It can further be shown 
that the characteristic polynomial now has a factor: 


N 
Biz") + —A(z™), 
bo 
so that ) can be considered a root-locus parameter; for plants with a stable A polynomial, a large enough 
value of \ ensures loop stability even for nonminimum-phase plants. Unfortunately for self-tuning 


applications, a prior “good” value of ). needs to be known; nevertheless, \ can be used to trade off output 
variance against control activity. 


33.8 Pole-Placement (PP) Self-Tuning 


The problem with MV regulators if that they are unstable if the plant has a zero outside the unit circle, 
as the controller attempts to cancel the zero by having an unstable pole. Hence, there is a mode in the 
control signal that grows without limit. Therefore, alternative strategies such as GMV must be used for 
more complex plants; one such appeals to classical control design. 

Recall the root-locus approach: it is a method of analysis showing how the poles of the closed-loop 
transfer function vary as some parameter, such as the controller gain, is changed. Design considers some 
pole-zero structure of a controller and then fixes the gain to give “nice” closed-loop pole positions, such as 
large ©, for a desired value of ¢ such as 1/./2. An alternative procedure that is often used in discrete-time 
control is first to choose the desired pole positions and then back-calculate the appropriate controller. 

Suppose that the feedback controller of Figure 33.12 is: 


F(z~!)y(t)+ G(z~)u(t) =0, where go = 1, (33.73) 


which gives the control action as: 


u(t) = —gyu(t — 1) — gu(t— 1) —--» —foy(t) —fiy(t — 1) 
If the plant has the CARMA model: 
A(z )y(t) = z*B(z!)u(t) + C(z7!)e(t), 


where the dead time k is explicitly factored from the zero polynomial, the closed loop is given by: 


[A(z !)G(z7!) + 2 *B(z F(z y(t) = Cle) G(z7 e(t). (33.74) 
Suppose that the polynomials F and G are obtained by solving the Diophantine identity: 


AG+z *BF =CT(z~!), (33.75) 
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where T(z~') is a polynomial chosen by the designer of the form: 


T(z?) =1+ tz! + toz? fees [a _ ajz!), 
i 


with the a; corresponding to the desired closed-loop pole positions. (For hand calculation, we compare 
powers of z~! as before; in a computer, we use Euclid’s algorithm.) By substituting Equation 33.75 into 
Equation 33.74, the closed loop is given by: 


C(z!) T(z") y(t) = C(z")G(z elt), 


or: 
G(z71) F(z!) 
t) = ——~e(t), d t)=-—=—e(t). 33.76 
YO = Faye» and ult) =~ Feel (33.76) 
Hence, the closed-loop poles are given by the user-chosen polynomial T; as the B polynomial is not 
cancelled by this law, there is no longer any unstable mode in the control signal even if the plant is 
nonminimum-phase. As an example, consider again the system: 


(1 — 0.927!) y(t) = 0.5u(t — 2) + (1 + 0.7z7!)e(t), 


and choose T(z~!) = 1 — 0.5z7!, giving a relatively fast closed-loop pole at a = 0.5. Then the Diophantine 
identity of Equation 33.75 becomes: 


(1—0.92)(1t+- giz 14+---)427705(% +fiz1---) =(4+0.7271)0 — 0.5271). 


Comparing coefficients of increasing powers of z~! gives g; = 1.1 and fy = 1.28, all other coefficients 
being zero. Hence, the control law is given by 


u(t) = —[1.lu(t — 1) + 1.28y(t)], 
and in closed loop we have from Equation 33.76: 


a 1+1.1z7} 


y(t) = {=o and u(t)= pe See a 


1—0.5z71 


It can be shown that the output variance due to this control law is 5.60”, roughly double the MV result: 
you don’t get something for nothing. 

It is interesting to see that pole assignment (for this regulator case) can also be self-tuned using 
RLS without knowing or estimating the polynomial C. To see why this is so, let A(z~!) and B(z~!) 
be the solutions to a new Diophantine identity that does not include the C polynomial (compare with 
Equation 33.75): 


AG(z7!) + 27*BF(z7!) = T(z7?). (33.77) 


The polynomials F and G provided as “input” to this identity are those obtained by using the previous 
identity of Equation 33.75 with the “true” plant polynomials A, B, C. Let us take our example, giving the 
new Diophantine identity of Equation 33.77: 

A(1 + L.1z7!) +.277B1.28 = (1 — 0.5271), 


giving A = 1— 1.6z~!,B = 1.6 x 1.1/1.28 = 1.375. 
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Consider operating the plant in closed loop using the “correct” controller —F/G as computed by 
Equation 33.75; then the sequence: 


AG +z ‘BF 


Te) e(t) = e(t), (33.78) 


A(z” !)y(t) — 2 *B(z"" u(t) = 
using the closed-loop behavior from Equation 33.76 and then the second Diophantine identity of Equa- 
tion 33.77. This result gives the key idea: when in closed loop with the right PP controller, there exists a 
relationship between the input and output signals given by the new polynomials A and B such that the 
error term is just the white driving noise. 

Equation 33.78 shows that RLS can be used to get unbiased estimates of A and B, as the right-hand 
side noise term is uncorrelated. To estimate the parameters, simply choose the data vector to be 


x(t) =[—y(t — 1), —y(t — 2),...;u(t — 1), u(t — 2), ...]' 


and with model output y(t). Hence, if the data sequences follow those of the pole-placed closed loop of 
Equation 33.76, then LS simply gives 6 — [A,B]. If these estimates are placed as input to the second 
Diophantine identity of Equation 33.77, we can use the identity to recompute the controller F 
and G polynomials; ie., the procedure of going [A, B, C, T] > [F, G] via 33.75 — [A,B] via RLS > 
LF, G] via 33.77 is self-consistent. 

Hence, the self-tuned version of PP goes through the following steps: 


1. Use RLS to obtain estimates A and B from the model Ay(t) =z *Bu(t)+ e(t) (ie, no C 
estimated). 

2. Resolve the Diophantine identity of Equation 33.77 at each sample for user-chosen T (using as 
input the estimates A and B) to obtain F and G, by equating powers of z~! as usual. 

3. Assert the control Fy(t) + Gu(t) = 0, or u(t) = —[F/G]y(t). 


As with self-tuned MV, the above argument is simply to give plausibility to the approach; in practice, 
the procedure converges provided that there is a solution to the Diophantine identity of Equation 33.77 
(which is not possible if there are common roots in the estimated model polynomials). 


33.9 Long-Range Predictive Control 


For practical applications, an adaptive controller must be robust against the prior assumptions made 
about the plant to be controlled. For example, we must choose sometimes arbitrary values for the degrees 
of the estimated polynomials and have no assurance that, for all t, our model is of neither too high nor 
too low an order compared with the true plant model within the bandwidth of our closed loop. The 
estimates may be affected over periods of time by disturbances not fully captured by our assumptions 
about C(z~!). There may be occasions when there are common factors in the estimated TFs. The dead 
time of the plant may vary so that the k assumed in k-step-ahead prediction for MV control is not correct 
(so that the “true” value of go is zero). Fractional dead time and fast sampling might cause the model to 
become nonminimum phase. Hence: 


Minimum-variance: We might get instability by assuming too small a value of k, or if the plant is 
nonminimum phase. 

Pole placement: There is no solution to the Diophantine identity if there are common factors in the 
estimated A, B polynomials. 


Results of controlling a time-varying plant with an adaptive MV algorithm with a forgetting factor of 
0.98 are shown in Figure 33.13. The plant changes its order n and delay k at various stages during the run. 
When the actual plant delay exceeds the assumed delay of the algorithm, MV goes unstable. Similarly, 
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FIGURE 33.13 Simulation of the adaptive MV control of a dynamically time-varying plant. 


it is found that with adaptive PP, the control goes unstable if the assumed model order is greater than 
the actual model order. With a fixed PID control the initial results are good, but instability sets in when 
the plant changes its dynamics. In practice, good engineering design using prior knowledge and proper 
choice of sample interval (e.g., at most 0.1 times the open loop rise time) can give good results with MV 
and PP. On the other hand, you can be unlucky. 

Long-range predictive control (LRPC) is a more modern approach that overcomes many of the above 
problems and has many extra features, which makes it useful in applications. The basic idea is to predict the 
future output as the sum ofa free response (based on past known data) and a forced response depending on 
current and future control actions, as shown in Figure 33.14. Consider the noise-free incremental model: 


A(z7!)Ay(t) = B(z~')Au(t — 1), (33.79) 


or: 


y(t) = y(t — 1) — ay Ay(t— 1) —--- +b, Au(t—1)+4+---. (33.80) 
Consider using this model to give the prediction p(t + 1) of the free output response based on maintaining 
the control signal equal to the previous value u(t — 1): 
pl(t+1) =y(t)— a Ay(t)—---+boAut—1)+---, 
as by assumption Au(t) = 0. Continuing the iteration further: 
p(t+2) =p(t+1)— a, Ap(t+1)—---+b3Au(t—1)+---, 


where the term Ap(t + 1) = p(t+ 1) — y(t), the difference between the prediction and the available data 
y(t). After some stage all the Au(t — 1) terms drop out as the polynomial B has been exhausted, leaving 
the iterations: 

p(t +i) = p(t +i-—1)—aAp(t+i-1)—a,Ap(t+i-2)----. (33.81) 


Hence, the predictions can be expressed verbally as “iterate the plant equations forward in time, assuming 
current and future control increments (moves) are zero, and using existing old Au, Ay to initialize the 
data.” 
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FIGURE 33.14 Long-range predictive control. 


Now consider the forced component of response, with the control input being a set of moves 
Au(t), Au(t+1),..., which are to be determined by the algorithm. This is simply a series of “steps” 
so that the output is just the superposition of a set of step responses {s;}, giving the total response as: 


y(t +1) = 5, Au(t) + p(t + 1) 
y(t +2) = s;Au(t+ 1) +s.Au(t) + p(t+ 2) 


Wtt+j) = s,Au(t +j—1) +s. Au(t+j—2)+---+5Au(t) + p(t +)) 


Hence, we can collect N such equations into vector-matrix form: 


y = GAu+ p, (33.82) 
where: 
y=D(t+ 1), y(t+2),..., y(t +N) 
Au=[Au(t), Au(t+1),..., Au(t+ N —1)] 
p=(p(t+ 1), p(t+2),...,p(t+N)]’ 
and: 
S] 0 0 
S2 S] 0 


SN SN-1 <*°* SI 
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Suppose that we had a “future set point” sequence {w(t + 1) --- w(t + N)} available at time f. In robotics, 
this could be the required future trajectory, whereas in process control we would normally assume the 
future set point to equal the current value. Either way, we can collect the sequence into a vector and hence 
define a vector of “future system errors”: 


e=[w(t+1)—y(t+1), w(t +2) — y(t +2),...,wt+N)—y(t+N)Y, 


giving: 


e=w-—(GAu-+p). 


The only unknowns in this set of equations are the future controls, so we minimize S = )~ e(t + j)? over 
the predictions by the set of controls: 


Au = (G'G)_'G'(w — p). (33.83) 


This gives a set of “best” future control actions; we then use a receding-horizon strategy, which simply 
asserts the first of this sequence and repeats the whole calculation at each sample instant. Note, however, 
that G is an N x N matrix and we get a solution only if it is invertible, giving: 


Au=G |(w—p). (33.84) 


This solution is exact such that the control sequence would drive all the future system errors to zero. 
Good in theory; bad in practice as it would require excessive control signals. Moreover, what happens if 
the plant delay is, say, 2 so that s; = 0? The result is failure, as we cannot then invert G. 

How can we derive more equations than unknowns to let LS “smooth” out our future errors? We 
can make assumptions about what controls will be exerted in the future. Consider controlling a simple 
Type 0 plant. We could inject a large initial signal to get it moving and then maintain a constant control 
of sufficient size to get it to the final set point. This would mean that at the initial time only two control 
increments are considered. Hence, we take: 


Au = [Au(t), Au(t + 1),0,0,..., 0], 


so that now we have fewer unknowns than equations. In general we can allow only NU control increments 
to be nonzero (called the “control horizon”; see Figure 33.14) and will define the “future control increment 
vector” to be 


Au=[Au(t), Au(t+1),..., Au(t + NU — LJ. 


This means that our set of equations for e involve a nonsquare matrix G. 

One special case is where only one control increment is considered at time f; i.e, NU = 1. In essence 
we are asking, “What step change in control will minimize the sum-of-squares of the future system 
errors?” Suppose we make the prediction horizon N very large. Then if there were any steady-state error, 
the corresponding sum-of-squares would be large compared with errors accruing during the transient. 
The outcome is that the control step will be just the right size to make the steady-state error zero, and there 
will be just the same dynamics in closed loop as in open loop. This simple approach is called mean-level 
control (see Figure 33.15). Given that NU = 1, the matrix inversion is simple, as G’G is just >. o a 
scalar. 

Consider the case where N is not large. Then the initial transient errors become increasingly important 
in comparison with the steady-state error so that the initial control is made larger to reduce them; i-e., we 
obtain a faster response as N reduces. 
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FIGURE 33.15 Mean-level control. 


33.10 The Generalized Predictive Control (GPC) Cost Function 


We can include other ideas into the LRPC algorithm. For example, if it is known that the plant has a dead 
time, then it is clear that the current control u(t) cannot affect the future errors until the dead time is 
cleared. Hence, there is no point in putting e(f + 1)---e(t+k— 1) into the LS cost function. We might 
also want to have a mechanism for “trading” the cost of control (e.g., control variance) against output 
performance (i.e., error variance). Combining all these we get the generalized predictive control (GPC) 
cost-function: 


N2 NU 
Iopc(N1,N2,NU,%) = D0 e(t +i +20 Aue+i-1Y, 
i=N1 i=l 


where: 


N1 is the lower costing horizon 

N2 is the upper costing horizon 

NI — N2is the costing range 

NUis the control horizon; i.e., the “degrees of freedom” in the control 
X is the control weighting 


subject to the condition that assumed future control-increment sequence is zero after the control horizon. 
Using GPC we have the solution 


Au=(G'G+ 21) 'G'(w — p), (33.85) 
where the matrix G is now N2 — N1+ 1 by NU: 


SN1 SNI-I1 ott? 0 


SN1+41 SN1 SN1-1 
G => 


SN2 SN2-1 whee SN2—NU+1 


There are many possible combinations of the four “design parameters” (N1,N2, NU, d) but in practice, 
two main choices are made. The first is mean-level control as above: [k, large, 1,0], where “large” means 
about 10. The other is based on “dead beat” control where the idea is to attain the set point as rapidly as 
possible such that the error and all its derivatives become zero simultaneously. This can be shown to be 
achieved by the following choices of horizon: 


Nl=n, N2>2n-—1, NU=n and }=0, 
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FIGURE 33.16 Adaptive GPC control of a time-varying plant. 


where n is the largest power of z~! found in the model (including the extra because of the A = 1—z7!). 
Suppose we choose N2 = 2n — 1. Then G is x n, and, provided it is invertible the GPC solution gives 
J =0 (as the equations are solved exactly) and hence e(t + n) --- e(f + 2n— 1) are all 0. This means that 
the control is such as to make n successive future system errors zero, a prerequisite for state deadbeat 
control. 

It is not just the flexibility of GPC (see for example Soeterboek [10]) that gives it the power. It can 
handle realistic control problems that cannot be treated by other designs. Suppose that we know the 
control signal to be constrained (as with torque saturation in motors) or that some internal variable (such 
as temperature in a catalytic cracker) must not exceed some limit. In principle, we can use the prediction 
idea to test if any of our variables is likely to hit constraints and, hence, modify the control signal suitably. 
By running a plant nearer to constraints, we enhance quality and profitability, so LRPC is increasingly 
popular in industry. 

By connecting an RLS estimator to GPC on the plant as in Figure 33.13, using a prediction horizon of 
10 and a control horizon of 1 (approximating to mean-level control) we obtain the results of Figure 33.16: 
much better! 

The problem with “simple” GPC as presented above is the lack of stability proofs, except for some 
limiting cases such as mean-level control; with practical applications, mean-level control has been used 
most often and this has not been found to cause difficulties. However, in order to get stability guarantees, 
we can use infinite-horizon LQ (e.g., Bitmead et al. [2]) or adopt terminal constraints where the objective 
now is to minimize a cost function subject to the output y(t +j) exactly matching the set point over some 
future constraint range (see Clarke [3], and Mosca [9], for details). 


33.11 Robustness of Self-Tuning Controllers 


Applications of self-tuning control have to take into account the following practical problems: 


¢ Disturbances acting on a process are likely to be nonstationary and to have inconsistent behavior, 
e.g., during plant start-up. 
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¢ Certain patterns of disturbance can lead to the estimation of a poor plant model. 

e The dynamic order of the process is likely to be significantly greater than that assumed by the 
self-tuning control design. 

¢ Actuator nonlinearities (e.g., stiction) give unrepresentative small-signal behavior. 


Some of these difficulties (in particular, the problem of unmodeled dynamics) have been treated in 
some detail; others (e.g., nonlinear actuation) require careful attention to engineering detail. In general, a 
“good” self-tuner requires (1) a robust estimator; (2) a robust control design; and (3) “jacketing” software 
that takes into account practical process features. 

For improving the estimator, it is possible to add normalization and a dead zone. Normalization 
simply takes the regressor vector and produces a factor m(t — 1) = max{m, ||x(t — 1)||} and divides the 
prediction error Equation 33.23 by m(t — 1). A dead zone takes into account that the “true” prediction is 
of the form: 


H(t) = )- 6 xm(t — i) + D> Oxy (t — 1), 
1 1 


where x, is the “modeled” data input and x,, the “unmodeled” input. If, after normalization, the prediction 
error is smaller than a certain amount, then this is deemed to be due to the (assumed small) effect 
of unmodeled dynamics and the estimation is temporarily frozen. While there is a good theoretical 
background for this approach (see, for example, Mosca [9]), there is not much practical experience in 
how to choose an appropriate value for the dead zone parameter. 

Perhaps the most important advance for ensuring robustness is the reconsideration of the CARIMA 
model in the form: 
T(z?) 

A 


where T(z~!) is now considered to be an assigned design rather than an estimated polynomial for enhanc- 
ing estimation and control robustness. Multiplying up in Equation 33.86 gives: 


A(z~!)y(t) = B(z")u(t) + 


e(t), (33.86) 


A(z yl (t) = Bz!) (t) + e(t), (33.87) 


where the signals are given by yf = Ay/T, uf = Au/T;i.e., they are band-pass filtered by A/T. The effect 
of A is to remove dc-offset or constant-load disturbances; the effect of 1/T is to filter out high-frequency 
effects so that the estimation concentrates on low-frequency behavior. 

The above assumption about the disturbance also affects predictive controller design, as optimal 
predictions take into account the assumed model for the disturbance. For example, it can be shown 
that a predictive controller in closed loop satisfies: 


aT(z~!)w(t) = R(z~!)Au(t) + S(z7)y(8), (33.88) 
where the polynomials R, S satisfy the Diophantine identity: 
R(z A(z) A + B(z7))8(z7!) = Pez!) T(z74), (33.89) 


where P, gives the closed-loop poles (e.g., for mean-level control P, = A). Then the closed loop is stable 
for fixed estimated polynomials A, B provided that for all frequencies up to Nyquist: 


| | ; (33.90) 


where S is deduced from Equation 33.89 using the estimated parameters. Hence, the right-hand side of 
Equation 33.90 is known from the estimated model, and in particular, T can be chosen to ensure the 
bound is satisfied, particularly at high frequencies where the undermodeling problem arises; see [3] for 


Self-Tuning Control 33-35 


more details. It has been found by “benchmark” studies that good choice of T is highly significant. It is 
possible to consider different designs of T for the estimator and controller, but in practice it is convenient 
and near optimal to use the same polynomial. In general, the choice of T = A(1 — yz~!)™, where y is in 
the neighborhood of a dominant plant pole, is fairly effective. 


References 
1. Astrém, KJ. and Wittenmark, B., Adaptive Control, Addison-Wesley, Reading, MA, 1989. 
2. Bitmead, R.R., Gevers, M., and Wertz, V., Adaptive Optimal Control, Prentice Hall, Englewood Cliffs, 
NJ, 1990. 
3. Clarke, D.W., Ed., Advances in Model-Based Predictive Control, Oxford University Press, UK, 1994. 
4. Gawthrop, P.J., Continuous-Time Self-Tuning Control, Research Studies Press, Letchworth, UK, 1987. 
5. Goodwin, G.C. and Sin, K.S., Adaptive Filtering, Prediction and Control, Prentice Hall, Englewood Cliffs, 
NJ, 1984. 
6. Hang, C.C., Lee, T.H., and Ho, W.K., Adaptive Control, Instrument Society of America, Research 
Triangle Park, NC, 1993. 
7. Harris, C.J. and Billings, S.A., Eds., Self-Tuning and Adaptive Control: Theory and Applications, Peter 
Perigrinus Ltd., Stevenage, UK, 1981. 
8. Isermann, R., Lachmann, K.-H., and Drago, D., Adaptive Control Systems, Prentice Hall, Englewood 
Cliffs, NJ, 1992. 
9. Mosca, E., Optimal Predictive and Adaptive Control, Prentice Hall, Englewood Cliffs, NJ, 1995. 
10. Soeterboek, R., Predictive Control: A Unified Approach, Prentice Hall, Englewood Cliffs, NJ, 1992. 
11. Wellstead, P.E. and Zarrop, M.B., Self-Tuning Systems, Wiley, New York, 1991. 


34 


Model Reference 
Adaptive Control 


Bal) (nttotMcnOt icc cainieueccataranness 
34.2 MRAC Schemes 
Model Reference Control + Direct MRAC « 
Indirect MRAC « Robust MRAC 
DAS: ERA NGS cisssiocesicacasdaseuionsecrmeupaameenaiese 34-11 
Scalar Example: Adaptive Regulation + Scalar 
Example: Adaptive Tracking « Example: Direct 
MRAC without Normalization (n* = 1) « Example: 
Direct MRAC without Normalization (n* = 2) « 
Example: Direct MRAC with Normalization « 
Example: Indirect MRAC 


Petros Ioannou PERC PEBIBCS cece ci sscscsonelentcteares tease beens 34-18 
University of Southern California Partie RaW sca sacssichcssdereteiaeaansracesinaadcssnnaieae 34-19 


34.1 Introduction 


Research in adaptive control has a long history of intense activity involving debates about the precise 
definition of adaptive control, examples of instabilities, stability and robustness proofs, and applications. 
Starting in the early 1950s, the design of autopilots for high-performance aircraft motivated an intense 
research activity in adaptive control. High-performance aircraft undergo drastic changes in their dynamics 
when they fly from one operating point to another. These changes cannot be handled by constant gain 
feedback control. A sophisticated controller, such as an adaptive controller, that would be able to learn and 
accommodate changes in the aircraft dynamics was needed. Model reference adaptive control (MRAC) 
was suggested by Whitaker et al. [9] to solve the autopilot control problem. The sensitivity method and 
the MIT rule [18] were used to design the adjustment or adaptive laws for estimating the unknown 
parameters for the various proposed MRAC schemes. 

The work on adaptive flight control was characterized by “a lot of enthusiasm, bad hardware and 
nonexisting theory” [1]. The lack of stability proofs and the lack of understanding of the properties of 
the proposed adaptive control schemes, coupled with a disaster in a flight test [8] caused the interest in 
adaptive control in the late 1950s and early 1960s to diminish. 

The 1960s became the most important period for the development of control theory and adaptive 
control in particular. State-space techniques and stability theory based on Lyapunov were introduced. 
Developments in dynamic programming [11], dual control [13], and stochastic control in general and 
in system identification and parameter estimation [10] played a crucial role in the reformulation and 
redesign of adaptive control. By 1966 Parks [6] and others found a way of redesigning the MIT rule-based 
adaptive laws used in the MRAC schemes of the 1950s by applying the Lyapunov design approach. Their 
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work, even though applicable to a special class of linear, time-invariant (LTI) plants, set the stage for 
further rigorous stability proofs in MRAC for more general classes of plant models. 

The advances in stability theory and the progress in control theory in the 1960s improved the under- 
standing of adaptive control and contributed to a strong renewed interest in the field in the 1970s. On 
the other hand, the simultaneous development and progress in computers and electronics that made the 
implementation of complex controllers, such as the adaptive ones, feasible, contributed to an increased 
interest in applications of adaptive control. The 1970s witnessed several breakthrough results in the design 
of adaptive control. MRAC schemes using the Lyapunov design approach were developed and analyzed 
and the concepts of positivity and hyperstability were used to develop a wide class of MRAC schemes with 
well-established stability properties [4,5,12,18]. At the same time, parallel efforts for discrete-time plants 
in a deterministic and stochastic environment produced several classes of adaptive control schemes with 
rigorous stability proofs [14]. The excitement of the 1970s and the development of a wide class of adaptive 
control schemes with well-established stability properties was accompanied by a number of successful 
applications [15]. 

The successes of the 1970s, however, were soon followed by controversies over the practicality of 
adaptive control. As early as 1979 it was pointed out that the MRAC schemes of the 1970s could easily go 
unstable in the presence of small disturbances [12]. The nonrobust behavior of adaptive control became 
very controversial in the early 1980s when more examples of instabilities were published, demonstrating 
lack of robustness in the presence of unmodeled dynamics and or bounded disturbances [7,16]. These 
examples stimulated many researchers, whose objective was to understand the mechanisms of instabilities 
and find ways to counteract them. By the mid 1980s, a number of new redesigns and modifications 
were proposed and analyzed, leading to a body of work known as robust adaptive control. An adaptive 
controller is defined to be robust if it guarantees signal boundedness in the presence of “reasonable” 
classes of unmodeled dynamics and bounded disturbances as well as performance error bounds that are 
of the order of the modeling error. The work on robust adaptive control continued throughout the 1980s 
and involved the understanding of the various robustness modifications and their unification under a 
more general framework [2,12]. 

The solution of the robustness problem in adaptive control led to the solution of the long-standing 
problem of controlling a linear plant whose parameters are unknown and changing with time. By the end 
of the 1980s several breakthrough results were published in the area of adaptive control and in particular 
MRAC for linear time-varying plants [19] . 

The focus of adaptive control research in the late 1980s and early 1990s was on performance properties 
and on extending the results of the 1980s to certain classes of nonlinear plants with unknown parameters. 
These efforts led to new classes of MRAC-type schemes motivated from nonlinear system theory [3], as 
well as to MRAC schemes with improved transient and steady-state performance [2]. 

Adaptive control has been traditionally divided into two classes, the MRAC-type schemes and adaptive 
pole placement control (APPC) schemes. In MRAC both the poles and zeros of the plant are changed so 
that the closed-loop plant has the same input-output properties as those of a given reference model. In 
APPC only the poles of the plant are changed. In this chapter we concentrate on MRAC for continuous- 
time plants that attracted considerable interest in the literature of adaptive control. For information on 
APPC and discrete-time adaptive control, the reader is referred to [17] and [14]. 


34.2 MRAC Schemes 


Model reference adaptive control (MRAC) is derived from the model-following problem or model refer- 
ence control (MRC) problem. In MRC, a good understanding of the plant and the performance require- 
ments it has to meet allows the designer to come up with a model, referred to as the reference model, that 
describes the desired input-output properties of the closed-loop plant. The objective of MRC is to find 
the feedback control law that changes the structure and dynamics of the plant so that its input-output 
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Reference model 
—’ 
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Controller Uy Plant 


L__y| c (8%) G,(s) 


Vp 


FIGURE 34.1 The diagram shows the basic structure of model reference control (MRC). 


properties are exactly the same as those of the reference model. The structure of an MRC scheme for a 
LTI, single-input, single-output (SISO) plant is shown in Figure 34.1. The transfer function W,,,(s) of the 
reference model is designed so that, for a given reference input signal r(t), the output y,,(t) of the refer- 
ence model represents the desired response the plant output yp(t) has to follow. The feedback controller, 
denoted by C(0*), is designed so that all signals are bounded and the closed-loop plant transfer function 
from r to yp is equal to W,,(s). This transfer function matching guarantees that for any given reference 
input r(t), the tracking error e)(t), which represents the deviation of the plant output yp from the desired 
trajectory y, converges to zero with time. The transfer function matching is achieved by cancelling the 
zeros of the plant transfer function Gp(s) and replacing them with those of W,,(s) through the use of 
the feedback controller C(0*). The cancellation of the plant zeros puts a restriction on the plant to be 
minimum phase, i.e., have stable zeros. If any plant zero is unstable, its cancellation may easily lead to 
unbounded signals. 

The design of C(87) requires knowledge of the coefficients of the plant transfer function Gp(s). If * is 
a vector containing all the coefficients of Gp(s) = Gp(s, &*), then the controller parameter vector 07 may 
be computed by solving an algebraic equation of the form 


Q* = F(6*) (34.1) 


When 0* is unknown the MRC scheme of Figure 34.1 cannot be implemented, since 0* cannot be 
calculated using Equation 34.1 and is therefore unknown. One way of dealing with the unknown param- 
eter case is to use the certainty equivalence approach [14,17]. In this context, the certainty equivalence 
approach is to replace the unknown 6% in the control law with its estimate 0,(t) obtained using the direct 
or the indirect approach. The resulting control schemes are known as MRAC and can be classified as 
indirect MRAC, shown in Figure 34.2, and direct MRAC, shown in Figure 34.3. In indirect MRAC the 
controller parameter vector , is calculated at each time using the estimate of the plant parameter vector 
§* and the mapping defined by Equation 34.1. In direct MRAC the vector @, is adjusted directly without 
any intermediate calculations that involve estimates of @*. In this case the plant transfer function G,(s, 0*) 
is parameterized with respect to 97 to obtain Gp(s, 97), whose form is used to estimate 0F directly. 

Different choices of on-line parameter estimators lead to further classifications of MRAC. 


34.2.1 Model Reference Control 
Consider the SISO, LTI plant described by the vector differential equation 
Xp =ApXp+Bpup, xp(0) = xo 
Yp = Cp Xp (34.2) 


where xp € R"3 yp, Up € R'and Ap, Bp, Cp have the appropriate dimensions. The transfer function of the 
plant is given by 


(34.3) 
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FIGURE 34.2 The diagram shows the basic structure of indirect MRAC. 


where Z,(s), Rp(s) are monic polynomials and ky, is a constant referred to as the high-frequency gain. The 
reference model, selected by the designer to describe the desired characteristics of the closed-loop system, 
is given by 

Zm(s) 
™ Rm(s) 
where Z(s), Rm (s) are monic polynomials of degree Gm, Pm, respectively, and k,, is a constant. 

The MRC objective is to determine the plant input up so that all signals are bounded and the plant 
output yp tracks the reference model output yn as closely as possible for any given reference input r(¢) 
that is bounded and continuous. We refer to the problem of finding the desired up to meet the control 
objective as the MRC problem. 

In order to meet the MRC objective with a control law that uses signals that are available for measure- 
ment, we assume that the plant and reference models satisfy the following assumptions: 


¥m = Wn(s)r =k r (34.4) 


Plant Assumptions: 


P1. Z,(s) is a monic Hurwitz polynomial of degree mp, i.e., Zp(s) is a monic polynomial of degree mp 
that has all roots in the open left half s-plane 


Reference model Ym 
> 
W,,AS) 
Sy 
Controller Up 7 Plant ’ J 
- 4), Ca) | Gls. 0°) — Gls, 02) 
On-line <——_ 
L___»| parameter 
estimation of 0 }<—— r 
8. | 


FIGURE 34.3 The diagram shows the basic structure of direct MRAC. 
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P2. An upper bound n of the degree np of Rp(s) 
P3. the relative degree n* = Np — Mp of Gp(s) and 
PA. the sign of the high-frequency gain kp 


are known. 


Reference Model Assumptions: 


M1. Z,,(s), Rm(s) are monic Hurwitz polynomials of degree q,,,, Pm, respectively, where pm <n 
M2. The relative degree n7, = pm — 4m Of Win(s) is the same as that of Gp(s), ie. ny, = n* 


In addition to assumptions P1 to P4 and M1, M2, let us also assume that the plant parameters, i.e., the 
coefficients of Gp(s), are known exactly. Since the plant is LTI and known, the design of the MRC scheme 
is achieved using linear system theory. 

The MRC objective is met if up is chosen so that the closed-loop transfer function from r to yp has stable 
poles and is equal to W,,(s), the transfer function of the reference model. Such transfer function matching 
guarantees that for any reference input signal r(¢) the plant output yp converges to ym exponentially fast. 

We consider the feedback control law 


Up = OT w, +0370. + Oy +r = O77 w (34.5) 
where 6% = cee et él’; O= [wf , 5 Vp» r]", a(s) = [s"-2,5"-3,...,8,1]7 w= ps 2 = 
Ne Yp, Co» 95 € R'; 07,05 € R” | are constant parameters to be designed and A(s) is an arbitrary monic 


Hurwitz polynomial of degree n — 1 that contains Z,,,(s) as a factor, i.e., A(s) = Ao(s)Zm(s), which implies 
that Ao(s) is monic, Hurwitz, and of degree np = n — 1 — qm. The controller parameter vector 0% € R?” is 
to be chosen so that the transfer function from r to yp i.e., yp = G,(s)r given by 


- Ca RpZp.A? 
A[(A — Of 0)Rp — kpZp (0370 + 05.A)] 


G,(s) (34.6) 


is stable and is equal to Wi,(s) = kin aa for all s. 

Since the degree of the denominator of G,(s) is np + 2n —2 and that of Ry(s) is pm < n, for the 
matching equation 

a kplpi” vA 
7 sce 7 ay pes (34.7) 

to hold, an additional np + 2n — 2 — pm zero-pole cancellations must occur in G,(s). Now since Z,(s) is 
Hurwitz by assumption, and A(s) = Ao(s)Zm(s) is designed to be Hurwitz, it follows that all the zeros of 
G,(s) are stable and therefore any zero-pole cancellation can only occur in C~, the open left half of the 
complex plane. Choosing 


k 
(= (34.8) 
kp 
and using A(s) = Ao(s)Zm/(s) the matching Equation 34.7 becomes 
(A — OFT a)Rp — kpZp (03 a + 03.A) = ZpAoRm (34.9) 


Dividing both sides of Equation 34.9 by Rp(s), we obtain 


A-0O*'a—k wa FOA)=Z (2+4 =) 
1 PR 2 U3 Sep, PR 
‘P ? 


where Q(s) (of degree n — 1 — mp) is the quotient and kp A* (of degree at most np — 1) is the remainder 
of AgRmn/ Rp, respectively. Then the solution for 67, i = 1, 2,3 can be found by inspection, i.e., 


*7 a(s) = A(s) — Zp(s)Q(s) 
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Q(s)Rp(s) — Ao(s)Rm(s) 


037 a(s) + 0% A(s) = 
kp 


(34.10) 


where the equality in the second equation is obtained by substituting for A*(s) using the identity 


AoRm Ou kp A* 
Rp Rp 
The parameters 07, i = 1, 2,3 can now be obtained directly by equating the coefficients of the powers of 


s on both sides of Equation 34.10. Equation 34.10 indicates that, in general, the controller parameters 
07,1 = 1, 2,3 are nonlinear functions of the coefficients of the plant polynomials Z,(s), Rp(s) due to the 
dependence of Q(s) on the coefficients of Rp(s). When n = ny and n* = 1, however, Q(s) = 1 and the 0's 
are linear functions of the coefficients of Zp(s), Rp(s). 


Lemma 34.1: 


i. Let the degrees of Rp, Zp, A, Ag and Ry, be as specified in Equation 34.5. Then the solution 0. a 
[o*7, 637, 637] of Equation 34.9 or Equation 34.10 always exists. 
ii. In addition, if Rp, Zp are coprime and n = np, then the solution 0. is unique. 


The proof is based on the solution of certain Diophantine equations and is given in [17]. 

It can be shown that the control law (Equation 34.5) with 6% calculated from Equations 34.8 and 34.10 
guarantees that the closed-loop plant is stable and the tracking error e) = yp — yn converges exponentially 
to zero for any given bounded reference input r. 


34.2.2 Direct MRAC 


A direct MRAC scheme is formed by combining the control law (Equation 34.5), with @* replaced by its 
estimate 0,(t) at time f, ie., 


Up = 02 (tw (34.11) 


with an adaptive law that generates 0,(t) at each time t. 

The estimate 0,(t) of 0* is generated by first obtaining an appropriate parameterization of the plant in 
terms of 6* and then using parameter estimation techniques to form the adaptive law for 0,(t). Such a 
parameterization is developed by using the plant and matching equations to obtain 


e1 = Win(s)p* (up — O*" w) (34.12) 


where e} = Yp — Ym, and p* = 1/c} . Using Equation 34.12 a wide class of adaptive laws may be developed 
to estimate 0*, p*. The adaptive laws may be split into two major classes; those with unnormalized signals 
and those with normalized signals leading to direct MRAC without normalization and direct MRAC with 
normalization. 


34.2.2.1 Direct MRAC without Normalization 


The derivation and complexity of the MRAC scheme depends on the relative degree n* of the plant. 
For n* = 1 we choose the transfer function W,,(s) of the reference model to be strictly positive real 
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(SPR) [17,18]. Substituting for u,» = 6! (t)w in Equation 34.12 we obtain the error equation 
§ p=, q q 
€1 = Wals)p*02 w (34.13) 


where 6, = 0, — 0* is the parameter error. Since W,,(s) is SPR we can use the SPR-Lyapunov design 
approach [17] to generate the adaptive law 


8, = —Te, wsgn(p*) (34.14) 


where 
r=r'so 


which, together with Equation 34.11 forms the MRAC scheme. The adaptive law (Equation 34.14) is 
chosen so that a certain positive definite function V of the error states of Equation 34.13 and Equation 34.14 
is a Lyapunov function with the property 
dV 
—< 
dt ~ 
which implies uniform stability and with additional arguments, e;(t) > 0 as t > ov. 
When n* = 2, W,,(s) cannot be designed to be SPR because of assumption M2 and the fact that a 
transfer function of relative degree 2 cannot be SPR. In this case we rewrite Equation 34.12 as 


0 


€1 = Winls)(s +p) p* [uy — 0 wf] 


where up = spp Of = po and p > 0 is chosen so that Winls) = Wn(s)(s + p) is SPR. If we choose 
up = OF wf then 
€1 = Wmn(s)p* (02 we) 


which has the same form as Equation 34.13 and leads to the adaptive law 


9, = —Tejwesgn(p*) (34.15) 


The adaptive law (Equation 34.15) is generated by using up = OF wr. Since ug = pp the control input 
Up has to be chosen so that up = OF wr. We have 


Up = (s4 p)0 wr =O w4 OF wy 


where the second equality is obtained by treating s as a differential operator. Since 0, is given by Equa- 
tion 34.15, the MRAC scheme when n* = 2 is given by 


Up = 020+ wf (34.16) 
6.= —Tejwesgn(p*) 
When n* = 3 we can use the same procedure as in the case of n* = 2 to obtain the error equation 
e1 = Wm(s)(s+ pi)(s + p2)p* [uy — 027] 
where 


1 1 
= GG ap GG 


a 


and pj, p2 > 0 are chosen so that Wins) = Win(s)(s + p1)(s+ pz) is SPR. In this case we cannot choose 
uf = 6! @ since such a choice will have to include second derivatives of 0, in the expression for Up that are 
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not available for measurement. We go around this difficulty by using the nonlinear tools “backstepping” 
and “nonlinear damping” as explained in [3,17] to obtain the MRAC scheme 


Up = OF +62 1 — (s + p2)a0(7 1°) r9 (34.17) 
79 = —[p1 + a0(67T'o)7Iro + 7 Pee sgn(p*) 
6, =—Terdsgn(p*), p= yerro 


1 1 
= ®, € =e 
oy niet 1=@1 hat 


TO 


where do > 0 is a design constant and Wrls) = 1/(s+ qo). For n* > 3 the procedure is the same, but it 
leads to much more complex MRAC schemes. 

The above MRAC schemes guarantee that all signals are bounded and the tracking error converges to 
zero. If r is sufficiently rich then the MRAC scheme for n* = 1, 2 guarantees exponential convergence of 
the parameter error 0, and that the tracking error e goes to zero. For n* > 3 the convergence of 0, to 
zero is asymptotic [17]. 


34.2.2.2 Direct MRAC with Normalized Adaptive Laws 


This class of MRAC schemes dominated the literature of adaptive control due to the simplicity of their 
design as well as their robustness properties in the presence of modeling errors. The adaptive laws of these 
schemes are driven by a normalized error signal that “slows” down adaptation and improves robustness 
with respect to plant uncertainties. For this reason they are referred to as normalized adaptive laws. The 
MRAC law up = 67 w in Equation 34.11 remains unchanged and the parametric model (Equation 34.12) 
is used to generate the adaptive law for 0;. The parametric model (Equation 34.12) may be rewritten in 
various other forms, giving rise to a wide class of adaptive laws. For example, we can rewrite Equation 34.12 
as 

Z= Ot b> (34.18) 


where z = Wy(s)up, bp = [Win(s)o?, Wm(s)os, Win(s)¥p» Yp] or 
e1 = p* (up — 02") (34.19) 


where uf = Winls)up, > = Wn(s)o. Equation 34.18 is obtained by first rewriting Equation 34.12 as 
Vp —Ym = 0*[z— 0 hp — cX¥m + cXyp] and then using the identity p*c* = 1. 
Using Equation 34.12 and the SPR-Lyapunov approach [17], we have the adaptive law 


0, = —Tedsgn(p*) p= yek& (34.20) 
where 


€=e,—e|— WmL(en2) 
@1 = Wm(s)L(s)[ (up — 02 0)] 
g=up—O0b, b=L\s)o 
up = L~\(s)up, n =o ot+ uf 


L(s) is chosen so that WL is SPR and proper, and L~!(s) is proper and stable. 
Using Equation 34.19 and the gradient method [17] we have 


0. = —Tedsgn(p*) p= yeé& (34.21) 


where ¢, $, € are the same as in Equation 34.20 with L7!(s) = Win(s). 
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Using Equation 34.18 we can generate a wide class of adaptive laws using the gradient method with 
different cost functions as well as least squares [17]. The gradient algorithm is given by 


0. = Pr[Pedp] (34.22) 
and the least squares is given by 
6 = Pr[Pedp] 
fom - oe (34.23) 
m 
where €= — ZS OF bp» m =1+ 5 dp and Pr[.] is the projection operator that constrains co(t), 


the estimate of cp, to satisfy |co (t) | > Cm, Vt > 0, where cy, > 0 is alower bound for |cj |. The Pr(.) operator 
sets P = 0, when |co(t)| = cm and Cp < 0. The projection is used to guarantee that 1/(co(t)) is bounded 
for all t > 0, a property that is used in the stability analysis of the MRAC scheme (Equation 34.11) with 
8, generated by Equation 34.22 or 34.23. For the implementation of projection we require the knowledge 
of cy, a lower bound for |c5| and the sign of cj. 

The control law (Equation 34.11) with any one of the adaptive laws (Equation 34.20, 34.21, 34.22, or 
34.23) forms a direct MRAC scheme. As shown in [17] these schemes guarantee signal boundedness and 
convergence of the tracking error to zero. If, in addition, the reference input r(f) is sufficiently rich of 
order 2n then both the parameter and tracking errors converge to zero. The rate of convergence in the 
case of Equation 34.11 with Equation 34.22 or 34.23 is exponential, whereas for the case of the MRAC 
scheme (Equations 34.11 and 34.20, or 34.11 and 34.21), the convergence is asymptotic. 


34.2.3 Indirect MRAC 


In indirect MRAC the controller parameter vector 0,(t) in the control law (Equation 34.11) is calculated 
at each time t using the estimates of kp and of the coefficients of Zp(s), Rp(s) that are generated using 
an adaptive law. The calculation of 0,(t) is achieved by using the mapping defined by the matching 
Equations 34.8 and 34.10. 

As in the direct MRAC case the adaptive laws for the estimated coefficients of Zp(s), Rp(s) could be 
normalized or unnormalized. We concentrate on the normalized adaptive laws and refer the reader to [17] 
for results using unnormalized adaptive laws. 

The adaptive law for estimating kp and the coefficients of Zp(s), Rp(s) is generated using the parametric 
plant model 


z= a (34.24) 
where 
T 
s” 6 Oy, 1 (S) —al'_4(s) 
Z=>——)p> = Up, 
Apis)? Ap(s) ” Ap(s) “? 
OF =[0,...,0, Dinsp spelt € R*", py =[bm—1,-.-b9]! and po = [ay-1,..., do]! are the coefficient vec- 


tors of kp[Zp(s) —s™], Rp(s) —s”, respectively, Ap(s) is an nth order monic Hurwitz polynomial, 
an—1(s) = [s"71,...,5, 1! andby = kp. 

Using Equation 34.24 the estimate p(t) of @; may be generated using adaptive laws that are based on the 
gradient or the least squares methods. The controller parameter vector 9,(t) = [or (t), of (t), 03(t), cg(t)]2 
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is calculated from 6,(t) = [0,...0, kps Dis pe ]', the estimate of 95 at each time t, as follows: 


km 
colt) = = 
P 
81 On2(8) = A(s) — Zp(s.#) ¢ Qs.) (34.25) 
1 A aA 
07 On—2(s) + O3.A(s) = 7 IQs, t) @ Ry(s, t) — Ao(s)Rm(s)] 
P 
where 
Q(s, t) = quotient of Aols)Rm(s) 
Rp(s, t) 


Zisty = kps™ + pi om—1(s) 
Rp(s, t)=s"+ pl on—1(s), aj(s) =[s',s-},...,5,1]7 


and A e B denotes pointwise in time multiplication. From Equation 34.25 it is clear that the adaptive law 
for kp has to be modified using projection so that |kp(t)| = km = 0, where kj, > 0 is a lower bound for ky. 
As an example of an adaptive law consider the gradient algorithm 

Op) = Pr[Ted], (34.26) 


Z-Z 


where Pr[e] is the projection operator that guarantees Lal > km > 0 for all t> 0. The projection operator 
requires the knowledge of the sign of kp and the lower bound ky, of kp. The indirect MRAC scheme 
(Equations 34.11, 34.25, and 34.26) guarantees that 9,(t) given by Equation 34.25 exists and is bounded 
for any bounded estimate 9p, all signals in the closed-loop plant are bounded and the tracking error 
converges to zero with time. If, in addition, the reference signal r is sufficiently rich of order 2n then the 
parameter and tracking errors converge to zero exponentially fast [17]. 


34.2.4 Robust MRAC 


The MRAC schemes presented above are designed for the plant model (Equation 34.2) that is free of 
disturbances and unmodeled dynamics. In the presence of disturbances and/or unmodeled dynamics the 
above schemes may be driven unstable, as shown by several examples in [16]. These schemes can be made 
robust by modifying the adaptive laws using leakage, dead-zone, projections and their by-products [17]. 
For the MRAC schemes without normalization these modifications guarantee the existence of a region 
of attraction in which all signals are bounded and the tracking error converges to a smaller residual set. 
For the MRAC schemes with normalization the region of attraction becomes the whole space provided a 
special normalizing signal is used to bound from above all the modeling error terms that are required to 
be small in the low-frequency range. 

As an example, let us modify the direct MRAC scheme (Equations 34.11 and 34.20) for robustness 
using a leakage type of modification known as o-modification. We have 


Up = Ow 

6, = —Pedsgn(p*) — oT 0, 6 =—yee — onyp 
ny = 6" + uF +m, 

ms = —S9m; + us oye m,;(0) = 0 


where 01,62 > 0 are small positive constants, m, is the dynamic normalizing signal, and 89 > 0 is chosen 
so that the m; bounds from above any modeling error term in the plant. The rest of the signals are as 
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defined in Equation 34.20. If the above robust MRAC scheme is applied to the plant 
Vp = G,(s)(1 or Am(s))up 


where Aj,(s) isa multiplicative plant uncertainty with the property that A,,(s — 89/2) has stable poles, then 
for small Ago 4|| W(s — 89/2) Am(s — 80/2)lloo, Az SI] W(s — 80/2) Am(s — 80/2)ll2 where W(s — 89/2) is 
an arbitrary stable transfer function with stable W-!(s—8& /2) and W(s)A,(s) is strictly proper, we have 
signal boundedness for any finite initial condition. Furthermore, the tracking error has a mean square 
value of the order of Ago, Az. The details of the design and analysis of robust MRAC schemes are given 
in [17]. 


34.3 Examples 


In this section, we present several examples that illustrate the design and analysis of the MRAC schemes 
described in the previous sections. 


34.3.1 Scalar Example: Adaptive Regulation 
Consider the following scalar plant: 

x=ax+u, x(0)=xo (34.27) 
where a is a constant but unknown. The control objective is to determine a bounded function u = f(t, x) 
such that the state x(t) is bounded and converges to zero as t > oo for any given initial condition xo. Let 


—Am be the desired closed-loop pole where a, > 0 is chosen by the designer. In this case the reference 
model is 


Xm =—AmXm, Xm (0) = xo (34.28) 


Control law: If the plant parameter a is known the control law 
u=—k*x (34.29) 
with k* = a+ a, could be used to meet the control objective, i.e., with Equation 34.29, the closed-loop 
plant is 
x=—aAmx, x(0)=x9 
whose equilibrium x- = 0 is exponentially stable in the large. 
Since a is unknown, k* cannot be calculated and therefore Equation 34.29 cannot be implemented. A 


possible procedure to follow in the unknown parameter case is to use the same control law as given in 
Equation 34.29, but with k* replaced by its estimate k(t), i.e., we use 


u=—k(t)x (34.30) 


and search for an adaptive law to update k(t) continuously with time. 


Adaptive law: The adaptive law for generating k(t) is developed by viewing the problem as an on-line 
identification problem for k*. This is accomplished by first obtaining an appropriate parameterization 
for the plant (Equation 34.27) in terms of the unknown k”, as follows. 
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We add and subtract the desired control input —k*x in the plant equation 34.27 to obtain 


x=ax—k*x+k*x+u 


Since a — k* = —ay, we have 
X= —Anxtk*x+tu 
or : 
x= (u+k*x) (34.31) 
S+am 


Equation 34.31 is a parameterization of the plant equation 34.27 in terms of the unknown controller 
parameter k*. Since x, u are measured and a,, > 0 is known, many adaptive laws may be generated using 
Equation 34.31 as shown in [17]. 

Substituting for the control u = —k(t)x in Equation 34.31, we obtain the error equation that relates the 
parameter error k = k — k* with the estimation error €) =X, ie, 


€) = ame) — kx, €; =x (34.32) 


Due to Equation 34.31 the estimation error €,, which is defined as the error that reflects the parameter 
error k, is equal to the regulation error x. The error equation 34.32 is in a convenient form for choosing 
an appropriate Lyapunov function to design the adaptive law for k(t). We assume that the adaptive law is 
of the form : 

k=k=fi(e,x,u) (34.33) 


where f; is some function to be selected, and propose 


k2 


2 é 
Vee 
(€1,k) 2 + 2y 


for some y > 0 as a potential Lyapunov function for the system defined by Equations 34.32 and 34.33. 
The time derivative of V along the trajectory of this system is given by 


kf, 


V= am€i keyx+ 


Choosing fi, = ye1x, ie., 
k=yex= yx’, k(0) =ko (34.34) 


we have 
V= = Yer <0 


Analysis: Since V is a positive definite function and V <0, we have V € Lo, which implies that 
€1,k € Loo. Since €; = x, we also have that x € £5, and therefore all signals in the closed-loop plant are 
bounded. Furthermore, €; =x € £2, and €; =x € Loo which imply that €)(t) = x(t) > 0 as t > oo. 
From x(t) > 0 and the boundedness of k, we establish that k(t) > 0, u(t) > Oast > oo. 

We have shown that the combination of the control law (Equation 34.30) with the adaptive law 
(Equation 34.34) meets the control objective, in the sense that it forces the plant state to converge to zero 
while guaranteeing signal boundedness. 

It is worth mentioning that we cannot establish that k(t) converges to k*, i.e., that the pole of the closed- 
loop plant converges to that of the reference model given by —a,,. The lack of parameter convergence is 


less crucial in adaptive control than in parameter identification, since in most cases the control objective 
can be achieved without requiring the parameters to converge to their true values. The simplicity of this 
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scalar example, however, allows us to solve for ¢; = x explicitly, and study the properties of k(t), x(t) as 
they evolve with time. We can verify that 


2ce—* 


j= 0), = 
€1(t) =e nape €)=x 


(34.35) 
c[(c + ko — ae* — (c — kp + @)] 


(c+ ko — a)e* + (c—ko +a) 


k(t)=a+ 


where c* = yx + (ko — a)”, satisfy the differential equations 34.32 and 34.34 of the closed-loop plant. 
Equation 34.35 can be used to investigate the effects of initial conditions and adaptive gain y on 
the transient and asymptotic behavior of x(t), k(t). We have lim;.o.k(t)=a+c, ifc>0, and 
limt-+o9 k(t) =a—cifc <0, i.e, 


lim k(t) = koo = a+ [yx + (ko — a)? ]"/? 
> 00 


Therefore for x) £0, k(t) converges to a stabilizing gain whose value depends on y and the initial 
condition x9, ko. It is clear from Equation 34.35 that the value of ko is independent of whether ko is a 
destabilizing gain, i.e., 0 < ko < a, ora stabilizing one, i-e., ko > a, as long as (ko — a)* is the same. The 
use of different ko, however, will affect the transient behavior as it is obvious from Equation 34.35. In the 
limit as f + oo, the closed-loop pole converges to —(koo — a) which may be different from —a,. Since the 
control objective is to achieve signal boundedness and regulation of the state x(t) to zero, the convergence 
of k(t) to k* is not crucial. 


34.3.2 Scalar Example: Adaptive Tracking 


Consider the following first-order plant: 
x =ax+ bu (34.36) 


where a, b are unknown parameters but the sign of b is known. The control objective is to choose an 
appropriate control law u such that all signals in the closed-loop plant are bounded and x tracks the state 
Xm of the reference model given by 
bn 
as a r 


for any bounded piecewise continuous signal r(t), where am > 0, bm are known and x(t), r(t) are 
measured at each time t. It is assumed that a,,, b, and r are chosen so that x, represents the desired state 
response of the plant. 


Control law: In order for x to track x», for any reference input signal r, the control law should be chosen 
so that the closed-loop plant transfer function from the input r to the output x is equal to that of the 
reference model. We propose the control law 


u=—k*x+l*r (34.37) 


where k*, /* are calculated so that 


x(s) bi* bin Xm(s) 


= — = 34.38 
r(s) s—a+bk* stam r(s) 
Equation 34.38 is satisfied if we choose 
bin am ta 
= ke = 34.39 
b b (34.39) 


provided of course that b # 0, i-e., the plant is controllable. The control law (Equations 34.37 and 34.39) 
guarantees that the transfer function of the closed-loop plant, i.e., x(s)/r(s) is equal to that of the reference 
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model. Such a transfer function matching guarantees that x(t) = x(t), Vt > 0 when x(0) = x(0) or 
|x(t) — Xm(t)| > 0 exponentially fast when x(0) 4 x,(0), for any bounded reference signal r(t). 

When the plant parameters a,b are unknown, Equation 34.37 cannot be implemented. Therefore, 
instead of Equation 34.37, we propose the control law 


u=—k(t)x+l(t)r (34.40) 


where k(t), I(t) is the estimate of k*, I*, respectively, at time t, and search for an adaptive law to generate 
k(t), l(t) on-line. 


Adaptive law: As before, we can view the problem as an on-line identification problem of the unknown 
constants k*,/*. We start with the plant equation, which we express in terms of k*,/* by adding and 
subtracting the desired input term —bk*x + bl*r to obtain 

bin b 


x= r+ (k*x —[*r+u) (34.41) 
S+am S+aAm 


Since Xm = fay is a known bounded signal, we express Equation 34.41 in terms of the tracking error 


defined as €) = x — Xp, i.e. 
b 
€= (k*x —T*r+u) (34.42) 
S+am 


Substituting for u=—k(t)x+l(t)r in Equation 34.42 and defining the parameter errors k Ak 
k*,141—I*, we have 


€) = ame, + b(—kx +ir) 
gio ctgs (34.43) 


The development of the differential equation 34.43 relating the estimation error with the parameter error 
is a significant step in deriving the adaptive laws for updating k(t), I(t). We assume that the structure of 
the adaptive law is given by 

k=file,x,r, u) i= filer, x,17,u) (34.44) 


where the functions f(, f2 are to be designed. 
Consider the function 


2 k2 2 
Ven k= = a [bl + =I 
1 


where y1, Y2 > 0, as a Lyapunov candidate for the system (Equations 34.43 and 34.44). The time derivative 
V along any trajectory of the system is given by 


; bil 
V =—ame? — bkeyx + Bur Oi | le (34.45) 
2 


Since |b| = bsgn(b), the indefinite terms in Equation 34.43 disappear if we choose f; = y1€1xsgn(b), 
2 =—y2e€11rsgn(b). Therefore, for the adaptive law 


k= yieixsgn(b), i= —y2€ rsgn(b) (34.46) 


we have 
q 2 
V= Ame} 


Analysis: Treating x;,(t), r(t) in Equation 34.43 as bounded arbitrary functions of time, it follows that 
V is a Lyapunov function for the third-order differential equations 34.43 and 34.46 and the equilibrium 
€1e = 0, ke 0, 1, 0 is uniformly stable. Furthermore, ek, le Loo and e€; € £2. Since €; =x —Xm, 
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Xm € Loo, we also have x € Loo and u € Lo and therefore all signals in the closed-loop plant are bounded. 
Now from Equation 34.43 we have €; € Loo, which together with €; € £2, implies that €)(t) > 0, as 
t — oo. We have established that the control law (Equation 34.40), together with the adaptive law (Equa- 
tion 34.46) guarantees boundedness for all signals in the closed-loop system. In addition, the plant state 
x(t) tracks the state of the reference model x,, asymptotically with time for any reference input signal 
r which is bounded and piecewise continuous. These results do not imply that k(t) > k*, I(t) > I* as 
t — oo, ie., that the transfer function of the closed-loop plant approaches that of the reference model as 
t — oo. In order to achieve such a result, the reference input r has to be sufficiently rich of order 2. A 
sufficiently rich input is one that excites all the modes of the system [14,17]. For example r(t) =sinwt for 
some w # 0 is sufficiently rich of order 2 and guarantees the exponential convergence of x(t) to xm(t) 
and of k(t), I(t) to k*, I*, respectively. In general, a sufficiently rich reference input r(t) is not desirable in 
cases where the control objective involves tracking of signals that are not rich in frequencies. 


34.3.3 Example: Direct MRAC without Normalization (n* = 1) 
Let us consider the second-order plant 


kp(s + bo) 
= — 
YP Se +ajstay ? 


where kp > 0, bo > 0, kp, bo, a1, 49 are unknown constants. The desired performance of the plant is speci- 


fied by the reference model 
1 


s+] 
Using the results of Section 34.2.2 the control law is designed as 


Yn = r 


Oy = —20, + Up» @ (0) =0 
@2 = —2w2 + Vp» w2(0) =0 
Up = 0,01 +8202 + 03Yp + cor 


by choosing A(s) = s+ 2 in the general control law. The adaptive law is given by 
0. =-—Tejw, 9,(0) = 9% 
where e] = Vp — Yin 9¢ = [6),02,3,co]2 , o = [w1,02,¥p,r]7 andr =I is any positive definite matrix. 


Analysis: From Equation 34.12 we have that the tracking error e, satisfies 


1 «OT 


ey = 
: stl 


where p* = kp, 0. =0,—0%, ie, é)=—e, + kp0 tw. 
We choose the positive definite function 


then 
V=-ef + kp0z ey — kp0z eva =-e} <0 
Therefore, e;, 9, are bounded, i-e., e1, 0, € Loo and ej is square integrable, ie., e1 € Lo. Since ym, e1 € 


Loo, we have Vp € Loo and therefore w2 € Loo. Now 


1 (s* +.a,s + ao) 
= Up = 
s+2.? (s+2)kp(s+ bo)? 


1 
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Since bo > 0, i.e., the plant is minimum phase and yp € Loo,we have @) € Loo. Hence, w € Loo, which 
implies that up € Loo. Since e1, OF w € Loo, we have @1 € £Lo0, which together with e, € £2, implies that 
ei(t) > Oast > oo. 

For parameter convergence, we choose r to be sufficiently rich of order 4. As an example, we select 
r(t) = A; sin w;t + A2 sin wat for some nonzero constants A;, A2, 1, ©2 with w; 4 w2. 


34.3.4 Example: Direct MRAC without Normalization (n* = 2) 


Let us consider the second-order plant 


kp 
= ——*___y 
YP Se +ajstay ? 


where kp > 0, a1, ao are constants. The reference model is chosen as 


5 
m= oT 
m= 452 


Using the results of Section 34.2.2 the control law is chosen as 


@, = —20, + Up, @2 = —2W2 + Yp 
b=-o+0 
Up = 02 © — b' They 


where @ = [o1,02ypr]", €1 =Vp — Ym p = 1, A(s) =s +2and 5(s + 1)/(s+ 5)* is SPR. The adaptive law 
is given by 


6. =e 
where [ =I! > 0 is arbitrary and 0, = [61, 02, 93, col’. 
Analysis: From Equation 34.12 by substituting for up we have that 
€1 = Win(s)kp(0z w+ 82 w) 


or 
e1 = Wmn(s)(s+ IkpO2 > (34.47) 


Since Wy,(s)(s + 1) is SPR and kp > 0 we can establish using the Lyapunov-like function 


T ATP—-10 
e’Pe @0°T'@, 
v= +—- 5 kp 


where e is the state of a state-space representation of Equation 34.47 and P = P’ > 0 satisfies the 
Lefschetz-Kalman-Yakubovich lemma [17] that 


V< —cet 


for some constant c > 0. This implies that e1, 6. € Loo and e; € La. Proceeding as in the case of n* = 1 
we can establish that all signals are bounded and e;(t) + 0 as t > oo. For parameter convergence, the 
input r is chosen as r(t) = A; sin w ft + A2 sin wat for some Aj, Ar # 0,0) 4 wo. 
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34.3.5 Example: Direct MRAC with Normalization 


In contrast to the direct MRAC schemes without normalization the complexity of the design and analysis 
of direct MRAC schemes with normalization does not change with the relative degree of the plant. We 
demonstrate the design and analysis of MRAC with normalization using the first-order plant 


x =ax+ bu 


where a, b are unknown and b > 0. The closed-loop plant is required to be stable and the state x is required 
to track the state x,, of the reference model 


bm 
Xn = 
S+am 


for any given bounded reference input signal r. If a, b were known the control law 
u=—k*x+l*r (34.48) 


with k* = (a, +.)/b,I* = b,,/b could be used to meet the control objective exactly. Since a,b are 
unknown, we replace Equation 34.48 with 


u=—k(t)x+[(t)r (34.49) 


where k(t), [(t) are the on-line estimates of k*, 1*, respectively. We design the adaptive laws for updating 
k(t), I(t) by first developing appropriate parametric models for k*, /*. We can show that the tracking error 
e1 satisfies 


b 


— (—k* I* 
aa (—k*x+F*r)] 


y= 


which may be written in the form of Equation 34.19 in Section 34.2.2, ie., 


e, = (ur — 07") 


where 0% = [k*, }", 0 = Wrls\[—x, rl, up = a u. Using the gradient method and the fact that b > 0 
we have 


0, = —Ted, b = yer (34.50) 


where 6,, b are the estimates of 0*, b, respectively, 


E=up—0h, m=1t+o o+uy? 


The stability analysis of the MRAC examples is accomplished as follows. First we show that Equation 34.50 


guarantees that 6,, b, €,em € Lo ande, em, 6, be £2 independent of the choice of u and the boundedness 
of ¢, u, e;. These properties are then used to establish the boundedness of all signals in the control loop 
and the convergence of the tracking error e; to zero. The details of the analysis are given in [17]. 
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34.3.6 Example: Indirect MRAC 
Consider the following third-order plant: 


1 


Waa"? (34.51) 


Vp = 


where a is the only unknown parameter. The output y, is required to track the output of y,, of the 
reference model i 


Yn = Gao 


The control law is given by 


Up = 911 + 822 + 03yp + cor 


1 
up +0 21 (s (+)? 


eur? 


1 
Up + 912 ———— 
Pp alte a eho 


s 
(s+ 01)? 
where 6, = [9115 912; 9215 922, col! € R®. In direct MRAC, 9, is generated by a sixth-order adaptive law. In 
indirect MRAC, 6, is calculated from the adaptive law as follows. Using the results of Section 34.2.2 the 
estimate @ of the only unknown plant parameter a is given by 


a= Yaba€ 
_ £2 ~ of _ 
Tere Z=050, Z=Ypthp b2 
2 T 
Spel! eta _ [s*5s,1] 
>=[9);63] > = Gap P: 
je [seg 
2 GMP? 


where = = [0, 0, 1,4,0,0]", A(s) is chosen as A(s) = (s+1)?, Xp =[341,3A7,3971's a = [0, 0,0, 


1,0, 0]o = G aa) Gu? and y, > 0 is a constant. The controller parameter vector is calculated as cy = 1, 
67 [s, ie = (s+ hale ~~ Qs, t) 


05 [s, 1]? + 03(s + 1)” = Qs, t) @[s? + as7] — (s+ 2,)°(s +2)? 


where Q(s, t) is the quotient of (s + m1)2(s+ 2)3/(s? + as’). 
The example demonstrates that for the plant equation 34.50, the indirect scheme requires a first-order 
adaptive law, whereas the direct scheme requires a sixth-order one. 
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Englewood Cliffs, NJ, 1993. 


More information on the design and analysis of robust MRAC can be found in the books 1, 2, 3, and 6 given 


above. 


Details on MRAC for discrete-time plants can be found in the book 4 by Landau given above and in: 
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Adaptive Filtering Prediction and Control by G. Goodwin and K. Sin, Prentice Hall, Englewood Cliffs, 
NJ, 1984. 


For applications of MRAC the reader is referred to the following books. 
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Peter Peregrinus, London, 1981. 

Adaptive and Learning Systems: Theory and Applications, Edited by K. S. Narendra, Plenum Press, New 
York, 1986. 
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35.1 Introduction 


The design of autopilots for high-performance aircraft was one of the primary motivations for active 
research in adaptive control in the early 1950s. Aircrafts operate over a wide range of speeds and altitudes, 
and their dynamics are nonlinear and conceptually time-varying. For a given operating point, specified 
by the aircraft speed (Mach number) and altitude, the longitudinal nonlinear aircraft dynamics can be 
approximated by a linear model. As the aircraft goes through different flight conditions, the operating 
point changes. These changes cannot be handled by constant gain feedback control. Since the output 
response y(t) carries information about the state as well as the parameters, one may argue that in principle, 
a sophisticated feedback controller should be able to learn about the plant changes by processing the 
input/output (I/O) measurements (u, y) and choosing the appropriate controller from a list or design a 
new one in real-time. The real-time or on-the-fly selection or design of the controller is what distinguishes 
adaptive from nonadaptive schemes. Figure 35.1 illustrates this general adaptive control structure. The 
structure covers almost all classes of adaptive control. The idea is to process the I/O and possibly auxiliary 
measurements and decide what controller to use in real-time. Under this generic structure one can include 
gain scheduling where the real time controller design block is just a look-up table with a scheduler logic. 
In identifier-based schemes, this block includes a parameter estimator and the online calculation of the 
controller whereas in nonidentifier-based schemes, the block may consist of multiple models, stored 
controllers, and so on and an appropriate logic for selecting the right controller in real-time. Structures 
such as direct and indirect adaptive control also fall into this general feedback structure. 
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FIGURE 35.1 General adaptive control structure. 


The assumption that the plant is linear time-invariant with constant parameters, some or all of which 
could be unknown, was used in almost all the early work on adaptive control. It was not till robust- 
ness issues were resolved that controlling linear plants with unknown time-varying parameters became 
possible [1]. An important part of most adaptive control schemes is the online parameter identifier or 
adaptive law, which generates estimates of the unknown parameters to be used for calculating or updat- 
ing the controller parameters in real-time. The way the adaptive law is combined with the control law 
gives rise to different adaptive control structures. The class of adaptive controllers that employ online 
parameter estimators are labeled as identifier-based adaptive control schemes. If instead of a realtime, 
parameter identifier one could partition the parameter space into a finite set of regions for which a 
controller could be designed a priori, the problem then becomes that of identifying in real-time which 
of the controllers is the right one to be switched on. Similarly, a bank of possible models for the plant 
can be formed. For each model a controller is designed a priori. A switching logic then can identify 
which plant model is valid based on input output measurements, and therefore choose the appropri- 
ate controller from a given list. These schemes that do not involve a parameter estimator or adaptive 
law are referred to as nonidentifier-based. There is also the class of adaptive controllers which could 
be a combination of identifier-based and nonidentifier-based schemes. In the following sections we 
discuss these different structures and associated stability and performance results, after we present a 
brief history how these different adaptive control structures and schemes evolved during the past 50 
years. 


35.1.1 Brief History 


Research in adaptive control has a long history of intense activities that involved debates about the 
precise definition of adaptive control, designs based on heuristics, stability-based schemes, examples of 
instabilities, stability and robustness proofs, and applications. 

Starting in the early 1950s, the design of autopilots for high-performance aircraft motivated an intense 
research activity in adaptive control. High-performance aircraft undergo drastic changes in their dynamics 
when they fly from one operating point to another that cannot be handled by constant-gain feedback 
control. A sophisticated controller, such as an adaptive controller, that could learn and accommodate 
changes in the aircraft dynamics was needed. Model reference adaptive control (MRAC) was suggested 
by Whitaker and coworkers in [2] to solve the autopilot control problem. The sensitivity method and 
the MIT rule was used to design the adaptive laws of the various proposed adaptive control schemes. An 
adaptive pole placement scheme based on the optimal linear quadratic problem was suggested by Kalman 
in [3]. The lack of stability proofs and the lack of understanding of the properties of the proposed adaptive 
control schemes coupled with a disaster in a flight test caused the interest in adaptive control to diminish. 

The 1960s became the most important period for the development of control theory and adaptive 
control in particular. State-space techniques and stability theory based on Lyapunov were introduced. 
Developments in dynamic programming [4], dual control [5], and stochastic control in general, and in 
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system identification and parameter estimation played a crucial role in the reformulation and redesign of 
adaptive control. By 1966 Parks and others [6] found a way of redesigning the MIT rule-based adaptive 
laws used in the MRAC schemes of the 1950s by applying the Lyapunov design approach. Their work, 
even though applicable to a special class of LTI plants, set the stage for further rigorous stability proofs in 
adaptive control for more general classes of plant models in subsequent years. 

The advances in stability theory and the progress in control theory in the 1960s improved the under- 
standing of adaptive control and contributed to a strong renewed interest in the field in the 1970s. On 
the other hand, the simultaneous development and progress in computers and electronics that made the 
implementation of complex controllers feasible, contributed to an increased interest in applications of 
adaptive control. The 1970s witnessed several breakthrough results in the design of adaptive control. A 
Lyapunov stability-based design approach was used to design and analyze MRAC schemes in [7,8]. The 
concepts of positivity and hyperstability were used in [9] to develop a wide class of MRAC schemes with 
well-established stability properties. At the same time, parallel efforts for discrete-time plants in a deter- 
ministic and stochastic environment produced several classes of adaptive control schemes with rigorous 
stability proofs [10]. The excitement of the 1970s and the development of a wide class of adaptive control 
schemes with well-established stability properties was accompanied by several successful applications. 
The successes of the 1970s, however, were soon followed by controversies over the practicality of adaptive 
control. As early as 1979 it was pointed out that the adaptive schemes of the 1970s could easily go unstable 
in the presence of small disturbances [11]. The nonrobust behavior of adaptive control became very con- 
troversial in the early 1980s when more examples of instabilities were published demonstrating lack of 
robustness in the presence of unmodeled dynamics and/or bounded disturbances [12,13]. Understand- 
ing the mechanisms of instabilities and finding ways to counteract them became the focus of subsequent 
research studies. By the mid-1980s, several new redesigns and modifications were proposed and analyzed, 
leading to a body of work known as robust adaptive control. An adaptive controller is defined to be robust 
if it guarantees signal boundedness in the presence of “reasonable” classes of unmodeled dynamics and 
bounded disturbances as well as performance error bounds that are of the order of the modeling error. 
The work on robust adaptive control continued throughout the 1980s and involved the understanding of 
the various robustness modifications and their unification under a more general framework [14]. Global 
stability in the presence of unmodeled dynamics using various fixes and a dynamic normalizing signal 
was established in [15] for discrete-time systems. The use of the normalizing signal together with the 
switching o-modification led to the proof of global stability in the presence of unmodeled dynamics for 
continuous-time plants in [16]. 

The solution of the robustness problem in adaptive control led to the solution of the long-standing 
problem of controlling a linear plant whose parameters are unknown and changing with time. By the 
end of the 1980s several breakthrough results were published in the area of adaptive control for linear 
time-varying plants [1]. The focus of adaptive control research in the late 1980s to early 1990s was 
on performance properties and on extending the results of the 1980s to certain classes of nonlinear 
plants with unknown parameters. These efforts led to new classes of adaptive schemes, motivated from 
nonlinear system theory [17] as well as to adaptive control schemes with improved transient and steady- 
state performance [18,19]. New concepts such as adaptive backstepping, nonlinear damping, and tuning 
functions are used to address the more complex problem of dealing with parametric uncertainty in classes 
of nonlinear systems. 

In the late 1980s to early 1990s, the use of neural networks as universal approximators of unknown 
nonlinear functions led to the use of on-line parameter estimators to “train” or update the weights of 
the neural networks. Difficulties in establishing global convergence results soon arose since in multilayer 
neural networks the weights appear in a nonlinear fashion, leading to “nonlinear in the parameters” 
parameterizations for which globally stable on-line parameter estimators cannot be developed. This 
led to the consideration of single-layer neural networks where the weights can be expressed in certain 
parametric models that are convenient for estimation [20,21]. 


35-4 Control System Advanced Methods 


In the mid-1980s to recent years, several groups of researchers started looking at alternative methods of 
controlling plants with unknown parameters [22-27]. These methods avoid the use of online parameter 
estimators in general and use search methods to identify unknown parameters, multiple models to char- 
acterize parametric uncertainty, switching logic to identify the stabilizing controller, and so on. They were 
motivated from the fact that in identifier-based schemes the estimated plant has to be stabilizable at each 
instant of time in order for a controller to exist. Since there is no guarantee that the online estimator will 
generate estimates that correspond to a stabilizable plant at all times, it raises theoretical and implemen- 
tation issues that need to be addressed. Designing all the stabilizing controllers to cover all possible plant 
parameter changes a priori eliminates this problem and transfers it to the ability of identifying which one 
of the a priori designed controllers is the right one to use. One distinct advantage of these efforts which 
are currently continuing is that well-established techniques from robust control for LTI systems can be 
employed. In the following sections we will elaborate further on some of the most popular adaptive control 
methodologies. 


35.2 Identifier-Based Adaptive Control 


35.2.1 Direct and Indirect Adaptive Control 


An adaptive controller of the identifier-based class is formed by combining an online parameter estimator, 
which provides estimates of unknown parameters at each time instant, with a control law that is motivated 
from the known parameter case. The way the parameter estimator, also referred to as adaptive law, is 
combined with the control law gives rise to two different approaches. In the first approach, referred to as 
indirect adaptive control, the plant parameters are estimated online and used to calculate the controller 
parameters at each instant of time. This approach has also been referred to as explicit adaptive control, 
because the design is based on an explicit plant model. In the second approach, referred to as direct 
adaptive control, the plant model is parameterized in terms of the controller parameters that are estimated 
directly without intermediate calculations involving plant parameter estimates. This approach has also 
been referred to as implicit adaptive control because the design is based on the estimation of an implicit 
plant model. 

The principle behind the design of direct and indirect adaptive control is conceptually simple. It 
treats the parameter estimates at each instant of time as if they are the true ones. In indirect adaptive 
control, the parameter estimates are associated with a plant parameterization, such as the coefficients of 
its transfer function in the case of an LTI plant, and so on. Hence at each time, the estimated parameters 
can be used to generate an estimated plant. The estimated plant is then treated as the true one and is 
used to calculate the controller by following the same techniques as in the known parameter case. In 
direct adaptive control, the plant is parametrized with respect to the desired controller parameters and 
the estimator generates the estimated controller parameters directly. Again the estimated parameters at 
each instant of time are treated as the true ones. This design approach of treating estimated parameters 
at each time instant as the true parameters and using them to generate the controller parameters is 
called certainty equivalence and has been used to generate a wide class of adaptive control schemes by 
combining different online parameter estimators with different control laws. The idea behind the certainty 
equivalence approach is that as the parameter estimates converge to the true ones, the performance of 
the adaptive controller tends to that achieved by the desired controller in the case of known parameters. 
The parameter estimator or adaptive law has a multiplicative nonlinearity that makes the closed-loop 
system nonlinear and time-varying. Because of this, the analysis and understanding of the stability and 
robustness of adaptive control schemes are more challenging, since most of the practical control design 
tools incorporating robustness and performance specifications used for LTI systems are not applicable to 
time-varying and nonlinear systems and therefore to adaptive control. For example, we can no longer use 
pole location and gain or phase margins considerations to specify or analyze stability and performance. 
Instead nonlinear techniques need to be developed specifically for adaptive control. 
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35.2.2 Online Parameter Estimation 


The first step in the design of online parameter estimation algorithms is to lump the unknown parameters 
in a vector and separate them from known signals, transfer functions, and other known parameters in an 
equation that is convenient for parameter estimation. In the general case, the class of parameterizations 
of the form 

z= 07" 6, (35.1) 


where @* € R” is the vector with all the unknown parameters and z € R, $ € R” are signals available 
for measurement, are referred to as the linear static parametric model (SPM). The SPM may represent a 
dynamic, static, linear, or nonlinear system. Any linear or nonlinear dynamics in the original system are 
hidden in the signals z, » that usually consist of the I/O measurements of the system and their filtered 
values. 

Another type of parameterization is of the form 


z= W(q) (0*") ; (35.2) 


where z € R, @ € R” are signals available for measurement and W(q) is a known stable proper transfer 
function; q is either the shift operator in discrete time (ie., q =z) or the differential operator (q = s) 
in continuous time. We refer to Equation 35.2 as the linear dynamic parametric model (DPM). The 
importance of the SPM and DPM as compared to other possible parameterizations is that the unknown 
parameter vector 0* appears linearly. For this reason we refer to Equations 35.1 and 35.2 as linear in 
the parameters parameterizations. This property is significant in designing online parameter estimators 
whose global convergence properties can be established analytically. 

In some cases, the unknown parameters cannot be expressed in the form of the linear in the parameters 
models. In such cases, the parameter estimators algorithms based on such models cannot be shown to 
converge globally. Special cases of nonlinear in the parameters models for which convergence results exist 
is when the unknown parameters appear in the special bilinear form 


z=" (8*To+2), (35.3) 
z= W(@o" (66+ 21) (35.4) 


where z ER, $ € R”, and z €R are signals available for measurement at each time t, and p* ER, 
and 0* € R” are the unknown parameters. The transfer function W(q) is a known stable transfer func- 
tion. We refer to Equations 35.3 and 35.4 as the bilinear SPM (B-SPM) and bilinear DPM (B-DPM), 
respectively. 

In some applications of parameter identification or adaptive control of plants in state-space form whose 
state x is available for measurement, the following parametric model may be used: 


&=Anx t+ O*'O, (35.5) 
where A,, is a stable design matrix; ©* is an unknown matrix; ® = [x? ul], and x, u are signal vectors 
available for measurement. We refer to this class of parametric models as state-space parametric models 
(SSPMs). It is clear that SSPMs can be expressed in the form of DPMs and SPMs. Another class of 
state-space models that appear in adaptive control is 


x = Amx + BO*'S, (35.6) 


where B is also unknown but is positive definite, negative definite, or the sign of each of its elements 
is known. We refer to this class of parametric models as B-SSPMs. The B-SSPM model can be easily 
expressed as a set of scalar B-SPMs or B-DPMs. 


35-6 Control System Advanced Methods 


The parameter estimation problem can now be stated as follows: 


¢ Given the available measurements, generate 6(t), the estimate of the unknown vector 6”, at each 
time t. The parameter estimator or adaptive law updates @(t) with time so that as time evolves, 0(t) 
approaches or converges to 0*. Since we are dealing with online parameter estimation, we would 
also expect that if 8* changes, then the estimator will react to such change and update the estimate 
0(t) to match the new value of 6*. 


The online parameter estimators generate estimates at each time ¢, by using the past and current 
measurements of signals. Convergence is achieved asymptotically as time evolves. For this reason they 
are referred to as recursive parameter estimators to be distinguished from the nonrecursive ones, in 
which all the measurements are collected a priori over large intervals of time and are processed off-line 
to generate the estimates of the unknown parameters. Generating the parametric models (Equations 35.1 
through 35.6) is the first step in the design of the appropriate parameter estimators. The essential idea 
behind online estimation is the comparison of the observed system response z(t), with the output of 
a parameterized model Z(6; t) whose structure is the same as that of the plant model. The parameter 
vector 6(t) is adjusted continuously so that Z(0;t) approaches z(t) as t increases. Under certain input 
conditions, 2(9; t) being close to z(t) implies that 0(t) is close to the unknown parameter vector 6* of the 
plant model. The adaptive law is usually a differential equation whose state is 0(t) and is designed using 
stability considerations or simple optimization techniques to minimize the difference between z(t) and 
(0; t) with respect to 0(f) at each time t. 

For the SPM case, the estimation model has the same form as the SPM with the exception that the 
unknown parameter 6* is replaced with its estimate at time t, denoted by 0(t), that is, 


z=0' (t), (35.7) 


where Z is the estimate of z based on the parameter estimate 9(t) at time f. Since 6* is unknown, 
the difference 0(f) = 0(t) — 6* is not available for measurement. Therefore, the only signal that we can 
generate, using available measurements, that reflects the difference between 6(t) and 6* is the error signal 


; (35.8) 


which we refer to as the estimation error. m2 > 1 is a normalization signal designed to guarantee that 
/mz, is bounded. This property of m; is used to establish the boundedness of the estimated parameters 
even when 94 is not guaranteed to be bounded. A straightforward choice for m,; is m2 = 1+ ao! , a >0. 
If @ is bounded, we can take m2? = 1. Using Equation 35.7 in Equation 35.8, we can express the estimation 
error as a function of the parameter error 6, that is, 


c=. (35.9) 


We update 6(f) in a direction that minimizes a certain cost of the estimation error €. As an example, 
consider the cost criterion 


(z— OT oy 
8) = ————,, 35.10 
J(9) am (35.10) 
which we minimize with respect to 6 using the gradient method to obtain the adaptive law 
§ = —-I'VJ(6) = Ted. (35.11) 


The gradient algorithm (Equation 35.11) guarantees €, ems, be Lo MLo; and 0 € Lo. 
The procedure for estimating 6* in a linear model extends to the other parametric models: the details 
of which can be found in [28,29]. 
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35.2.3 Model Reference Adaptive Control 


MRAC has been one of the most popular approaches to adaptive control. The basic structure of an 
MRAC scheme is shown in Figure 35.2 for the indirect scheme and Figure 35.3 for the direct scheme. 
The reference model is chosen to generate the desired trajectory, y, that the plant output yp has to 
follow. The tracking error e1 = yp — ym represents the deviation of the plant output from the desired 
trajectory. The closed-loop plant is made up of an ordinary feedback control law that contains the plant 
and a controller C(9) and an adjustment mechanism that generates the controller parameter estimates 
6(t) online. 

MRAC schemes can be characterized as direct or indirect and with normalized or unnormalized 
adaptive laws (Figure 35.4). In direct MRAC, the parameter vector 9 of the controller C(9) is updated 
directly by an adaptive law, whereas in indirect MRAC, 0 is calculated at each time t by solving a certain 
algebraic equation that relates 9 with the online estimates of the plant parameters through some mapping 
F(-). In both direct and indirect MRAC with normalized adaptive laws, the form of C(8), motivated from 
the known parameter case, is kept unchanged. The controller C() is combined with an adaptive law (or an 
adaptive law and an algebraic equation in the indirect case) that is developed independently by following 
the techniques of Section 35.2.2. This design procedure allows the use of a wide class of adaptive laws that 
includes gradient, least-squares, and those based on the SPR-Lyapunov design approach [28,29]. 
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FIGURE 35.4 Classes of MRAC. 


For most classes of MRAC, the controller and the parameter estimator are of the form 


u(t) = 0" (t)o(t); (35.12) 
0(t) = Te(t)(t), (35.13) 
for the direct scheme and 
u(t) = 07 (t)o(t), (35.14) 
Op(t) = Pe(t) p(t), (35.15) 
O(t) = F(8p(t)), (35.16) 


for the indirect scheme. These schemes guarantee that all signals are bounded and the plant output tracks 
the output of the reference model asymptotically with time for any input command. Furthermore, if the 
input command is sufficiently rich in frequencies, then 0(t) converges to the true parameter value 6%. 

In order to meet the model reference control (MRC) objective we must assume that the plant transfer 
function is minimum phase. The minimum-phase assumption is a consequence of the control objective 
which is met by designing an MRC control law that cancels the zeros of the plant and replaces them with 
those of the reference model in an effort to force the closed-loop plant transfer function from r to yp to be 
equal to W,,,(s). Other assumptions such as the knowledge of an upper bound for the plant order or the 
knowledge of the relative degree may be relaxed at the expense of additional complexity in the control 
and adaptive laws. 


35.2.4 Adaptive Pole Placement Control 


The assumption that the plant is minimum phase, that is, it has stable zeros, is rather restrictive in many 
applications. For example, the approximation of time delays leads to plant models with unstable zeros. 
Furthermore, the desired properties of the plant to be controlled are often expressed in terms of desired 
pole locations to be placed by the controller. Consequently, there is no need to alter the zeros of the plant. 
If the plant satisfies the properties of controllability and observability, then a controller always exists to 
place the poles in the desired locations. These schemes are referred to as pole placement schemes and are 
applicable to both minimum- and nonminimum-phase LTI plants. The combination of a pole placement 
control law with a parameter estimator or an adaptive law leads to an adaptive pole placement control 
(APPC) scheme that can be used to control a wide class of LTI plants with unknown parameters. 

As in the MRAC case, the APPC schemes may be divided into two classes: the indirect APPC schemes 
where the adaptive law generates online estimates of the coefficients of the plant transfer function that 
are then used to calculate the parameters of the pole placement control law by solving a certain algebraic 
equation; and the direct APPC where the parameters of the pole placement control law are generated 
directly by an adaptive law without any intermediate calculations that involve estimates of the plant 
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parameters. The direct APPC schemes are restricted to scalar plants and to special classes of plants where 
the desired parameters of the pole placement controller can be expressed in the form of the linear or 
bilinear parametric model. The indirect APPC schemes, on the other hand, are easy to design and are 
applicable to a wide class of LTI plants that are not required to be minimum phase or stable. Because 
of this flexibility in choosing the controller design methodology (observer-based state feedback, linear 
quadratic, etc.) and adaptive law (least-squares, gradient, or SPR-Lyapunov type), indirect APPC is the 
most general class of adaptive control schemes. This class also includes indirect MRAC as a special case 
where some of the poles of the plant are assigned to be equal to the zeros of the plant to facilitate the 
required zero-pole cancellation for transfer function matching. 
In the indirect case, the APPC scheme is of the form 


u(t) = 07 (t)o(t), (35.17) 
A(t) = Pe(t)bp(t), (35.18) 
F(O(£), 8p(t)) = 0, (35.19) 


where the last equation must be solved for the controller parameters @(t) at each time f. It is a pole 
placement equation to be solved pointwise in time. Such an equation has a unique solution only if 
the estimated plant polynomials are strongly coprime at each time t. Such a strong condition cannot 
be guaranteed by the adaptive law without any additional modifications, giving rise to the so-called 
“stabilizability” or “admissibility” problem, which is the main drawback of indirect APPC. This drawback 
can be eliminated by modifying the indirect APPC schemes at the expense of adding more complexity. 
If the estimated plant polynomials are strongly coprime at each time f then all signals are bounded and 
the tracking error converges to zero asymptotically with time. The same result holds if we replace the 
gradient algorithm with any other adaptive law. Furthermore, if the reference trajectory is sufficiently 
rich in frequencies then convergence is exponential and we have parameter convergence of 0(t) to the 
true parameter value 6*. The proof involves the manipulation of the estimation error and control law 
equations to express the plant input up, and output yp in terms of the estimation error in an equation of 
the form 

X(t) = A(t)x(t) + B67 o + d, (35.20) 


where the state x is formed by the plant input up and output yp and their derivatives, d is a bounded vector, 
and 6 is the parameter error. The matrix A(t) has stable eigenvalues at each frozen time t by design as a 
result of the pole placement approach. Furthermore, the adaptive law guarantees that | A(t) | € Lp. These 
two properties are used to show that the homogeneous part of Equation 35.20 is uniformly asymptotically 
stable (u.a.s.). Using the properties of the £2 norm and Bellman-Gronwall lemma, we can establish 
boundedness of up, and yp and also 6° € La M Loo. The convergence of e; to zero follows by using the 
control and estimation error equations to express e; as the output of proper stable LTI systems whose 
inputs are in £2 M Loo. In the direct case and MRAC, the form of Equation 35.20 is the same with the 
exception that the matrix A is time-invariant with stable eigenvalues, implying that the homogeneous 
part is u.a.s. The rest of the analysis follows the same way as in the indirect case using the same tools. 
Similar results and techniques can be established for discrete time systems. A wide class of MRAC and 
APPC can be designed to meet the performance requirements of signal boundedness and convergence 
of the regulation or tracking error to zero. This speed of convergence is asymptotic unless the reference 
input has a sufficient number of frequencies in which case we also have parameter convergence. In most 
applications the use of sufficiently rich signals is not possible as it violates the tracking performance 
requirements. In such case, the estimated parameters may not converge to any constant values but 
continuously move on some manifolds that guarantee signal boudedness and convergence of the tracking 
error to zero. This ability of the adaptive control schemes to guarantee tracking performance without 
having parameter convergence to the true values is remarkable. The price paid, however, is that in the 
presence of uncertainties the schemes may lose stability as discussed in the next section. 
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35.2.5 Instability Phenomena in Adaptive Systems 


The adaptive laws and control schemes discussed in previous section are based on a plant model that is 
free of noise, disturbances, and unmodeled dynamics. These schemes are to be implemented on actual 
plants that most likely deviate from the plant models on which their design is based. An actual plant may 
be infinite dimensional, nonlinear and its measured input and output may be corrupted by noise and 
external disturbances. The effect of these discrepancies on the stability and performance of the schemes 
designed using idealized models is therefore of paramount importance from the practical point of view. 
Let us examine the effect of modeling errors using the following simple plant example: 


y(t) = u(t) + d(t), (35.21) 
where d is an external bounded disturbance. The simple estimator 
A(t) = ye(t)u(t), €(t) = y(t) — O()u(t) (35.22) 


is designed assuming that the external disturbance does not exist, that is, d = 0. In the ideal case (d = 0) 
we can establish that €, 8 € Lo, and¢ — Oast — oo. When d is different than zero these results cannot be 
guaranteed. More important is that the boundedness of the estimated parameters cannot be guaranteed 
either. In fact, we can find an input u such that e — 0 and |8| — 00 as t > oo. This rather strange 
phenomenon is better understood by analyzing the parameter error equation 


8(t) = —yu2(t)0(t) + yd(t)u(t) (35.23) 


obtained from Equation 35.22. 

The homogeneous part of Equation 35.23 is not guaranteed to be u.a.s for all possible bounded inputs u. 
Consequently, bounded input does not imply bounded output. This instability phenomenon where the 
estimated parameters drift to infinity is known as parameter drift. It is mainly due to the pure integral 
action of the adaptive law, which, in addition to integrating the "good" signals, integrates the disturbance 
term as well, leading to the parameter drift phenomenon. 

Another kind of instability is the so-called high-gain instability. Consider the plant transfer function 


1— 1 2 
ae eee (35.24) 
(s—a)(1+ps) s—a 1+ ps 


where jw is a small positive number which may be due to a small time constant in the plant. Let us now 
design an adaptive controller for the simplified first-order plant (j1 = 0) and use it to control the actual 
second-order plant, where 1. > 0. The adaptive control law 


u(t) = —k(t)y(t), k(t) = yy(t) (35.25) 


guarantees that when t = 0, u, k, y € Loo, and y > 0 as t > oo for all initial conditions. But when jt 4 0, 
for k(0) > 1/2 — a, we have u, k, y > oo as t > ov, indicating that the presence of modeling errors 
reduced the stability of the closed-loop system from global to local. 

The instability examples presented demonstrate that the adaptive schemes designed for ideal plants, that 
is, plants with no modeling errors may easily go unstable in the presence of disturbances or unmodeled 
dynamics. The lack of robustness is primarily due to the adaptive law which is nonlinear in general 
and therefore more susceptible to modeling error effects. The lack of robustness of adaptive schemes in 
the presence of bounded disturbances was demonstrated as early as 1979 [11] and became a hot issue 
in the early 1980s when several adaptive control examples are used to show instability in the presence 
of unmodeled dynamics and bounded disturbances [12,13]. It was clear from these examples that new 
approaches and adaptive laws were needed to assure boundedness of all signals in the presence of plant 
uncertainties. These activities led to a new body of work referred to as robust adaptive control. 
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35.2.6 Robust Adaptive Laws 


It turns out that the destabilizing effects of bounded disturbances and dynamic uncertainties can be 
counteracted by modifying the adaptive laws developed for the ideal plants using techniques such as 
leakage, dead zones, projection, signal normalization, and so on. One unique characteristics of these 
modifications is that they remove the pure integral action of the adaptive laws at all or some of the time. 
In this section we extend the results of Section 35.2.2 to a general class of parametric models with modeling 
errors that may arise in the online parameter estimation problem of a wide class of plants. If the objective 
is parameter convergence, then parameter drift can be prevented by making sure that the regressor vector 
is persistently exciting (PE) with a level of excitation higher than the level of the modeling error. In many 
applications, such as in adaptive control, the plant input is the result of feedback and cannot be designed 
to be sufficiently rich. In such situations, the objective is to drive the plant output to zero or force it 
to follow a desired trajectory rather than convergence of the online parameter estimates to their true 
values. It is therefore of interest to guarantee stability and robustness even in the absence of persistence 
of excitation. This can be achieved by modifying the adaptive laws of the previous sections to guarantee 
stability and robustness in the presence of modeling errors independent of the properties of the regressor 
vector 0. 

A class of robust modifications involves the use of a small feedback around the “pure” integrator in the 
adaptive law, leading to the adaptive law structure 


Q(t) = Pe(t)(t) — o/(t)P°(t)0(¢), (35.26) 


where o; > 0 is a small design parameter and l = I’! > 0 is the adaptive gain. The above modification is 
referred to as the o-modification or as leakage [12,28,30]. Different choices of 07(t) lead to different robust 
adaptive laws. The simpler choice is the fixed o-modification: 


o(t)=o>0, VWt>0. (35.27) 


Another possible choice of o;(t) leads to the switching o-modification: and Mo is known: 


0 if |9(¢)| < Mo 
o1 = (4 — i)" 59 if Mo < |0(t)| < 2Mo, (35.28) 
or if |6(t)| > 2Mo 


where qo > 1 is any finite integer, og > 0 is a design constant, and Mp is a known upper bound of |6*|. 
Another effective way to guarantee bounded parameter estimates is to use projection to constrain 
the parameter estimates to lie inside a bounded convex set in the parameter space that contains the 
unknown 0*. By requiring the parameter set to be bounded, projection can be used to guarantee that the 
estimated parameters are bounded by forcing them to lie within the bounded set. As an example, consider 
the set 
p={6|6"9—Mj <0}, (35.29) 


where Mp is chosen so that Mo > |6*|. The adaptive law with projection becomes 


if 0'0< M3 or 


Ted if 070 = MP 
6= and (Ied)'0 <0 (35.30) 
reer ee 
(1 a a) Ted otherwise. 


In the presence of modeling errors the static linear parametric model is of the form 


z= 0 +n, (35.31) 
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where 1 is the modeling error term. Let us now consider the estimation error 


€= = (35.32) 


The signal € that drives the adaptive law is a function of the good signal that depends on the parameter 
error 6, and of the modeling error 1. Large em? implies that 4 is large and that the effect of the modeling 
error 1 is small, and therefore the parameter estimates driven by € move in a direction which reduces 
6. When em, is small, however, the effect of 1 may be more dominant than that of the signal 67, and 
the parameter estimates may be driven in a direction dictated mostly by n. Thus, it seems reasonable 
to update the parameter estimate @ only when the signal 6" is large relative to the disturbance n and 
switch-on adaptation when 67 is small relative to the size of n. This method of adaptation is referred to 
as dead zone because of the presence of a zone or interval where 0 is constant, that is, no updating occurs. 
The adaptive law in this case takes the form: 


Ted if lems| > go 


35.33 
0 otherwise, ( ) 


(1) = | 
where go is a known upper bound of the normalized modeling error y/ms. In other words, the parameters 
are updated in the direction of the steepest descent only when the estimation error is large relative to the 
modeling error, that is, when |ems| > go. 

Another important modification that helps improve robustness and performance is the use of a dynamic 
normalizing signal in the adaptive law. The dynamic normalizing signal is designed to bound from above 
the modelling error signals and in some sense make sure that the speed of adaptation is slower than 
the speed with which the modeling error terms change. We demonstrate the design of the dynamic 
normalizing signal for the SPM with modeling error 


z(t) = O* p(t) + n(f), (35.34) 


where y(t) = Aj(s)u(t) + Az(s)y(t) in the case of an LTI plant with additive and multiplicative uncer- 
tainties. A;(s), A2(s) are strictly proper transfer functions with stable poles. Our objective is to design 
a dynamic normalizing signal m, so that n/m; € Loo. We assume that Aj(s), Ao(s) are analytic in 
}i[s] > —89 for some known 89 > 0. If we define ng = ||uzII23, + lve which can be generated by the 
differential equation 


las, 


iig(t) = —8ong(t) + u(t) +y7(t). (35.35) 


then it follows that m? = 1+ ng bounds |(t)| from above and 


In(Z)| 
—— S lAilles, + ll Aallas, » (35.36) 
ms 
7 1/2 
where ||Hll5 = Tk | he H(jw — §)| do| . The normalizing signal m,, used in the adaptive laws 


for parametric models free of modeling errors, is required to bound the regressor ¢ from above. In the 
presence of modeling errors, m; is chosen to bound both $ and the modeling error y from above for 
improved robustness properties. An example of such normalizing signal is 


m =14+n2+n4, (35.37) 


where ns is the static part and nq the dynamic one. Examples of static and dynamic normalizing signals 
are n? = 6! or n? = b! Pd, where P = P? > 0, and 


iig(t) = —Song(t) + 81(u*(t) +7 (4) (35.38) 
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or 
ng= ny,  in(t) = —Somi(t) + 81 (Ju(t)| + yO) (35.39) 
or 
ng = N25» Noo = $8; max (sep |u(t)|, sup po) : (35.40) 
t<t t<t 


Any one of the choices (Equations 35.38 through 35.40) can be shown to guarantee that n/m, € Loo. 
Since = H(s)[u y]’, for an appropriate transfer function H(s), the dynamic normalizing signal can be 
chosen to bound $ from above, provided that H(s) is analytic in R[s] > — i, in which case m? =l+ng 
bounds both 6 and y from above. 

The above adaptive laws are shown to be robust with respect to a wide class of plant model uncertainties 
and can be combined with control laws to generate robust adaptive control schemes. Let us consider the 
plant 


y(t) = Go(s)(1 + Am(s))u(t) + d(t), (35.41) 


where Go(s) represents the dominant or modeled part of the plant transfer function, A,,(s) is a multiplica- 
tive perturbation, and d(t) is a bounded disturbance, that is, |d(t)| < do. It is important to note that we 
can rewrite Equation 35.41 in the form of Equation 35.34. Suppose there exists a strictly proper transfer 
function W(s) analytic in R[s] > —89/2 and such that W(s)A,,(s) is strictly proper. The result is that 
there exists a 8* > 0 such that if cA? < 8*, where A, = ||W(s)Am(s)ll23, and c > 0 is a finite constant, 
then all the signals are bounded and the tracking error e) satisfies 


t+T 

Ss 1 e2(t) dt < co(A2 + d2 + fo) + a Vt>0 (35.42) 
for any T' > 0 and some positive constants co, ci, where fo = 0 in the case of switching o-modification 
and projection, and fo > 0 in the case of fixed o-modification (f9 =o) and dead zone (fo = go). Due 
to the presence of modeling errors we no longer have exact convergence of the tracking error to zero. 
Rather the tracking error is of the order of the modeling error in the mean square sense. This, however, 
does not guarantee that the tracking error will be bounded from above by the modeling error bound 
at all times. In fact, simulations show that a phenomenon known as “bursting” could occur. That is the 
tracking error converges to a small value where it stays for some time before it bursts to large values and 
then again converges back to the small steady-state value. Condition (Equation 35.42) does not exclude 
such phenomena. One explanation of the “bursting” phenomenon is that when the signals are small the 
modeling error becomes dominant and could force the estimated parameters to drift to the unstable 
region. The plant output then grows causing the good signals to be come dominant which then force the 
estimated parameters to move to a region that corresponds to a smaller tracking error. This phenomenon 
can keep repeating itself and is a major drawback. One way to get rid of it is to use PE signals but this is 
not practical as such signals will interfere with the performance objective. A more practical approach is 
to use a dead zone to switch-off adaptation when convergence to steady-state values is achieved [1]. 


35.3 Nonidentifier-Based Adaptive Control 


The stabilizability issues encountered in the case of indirect adaptive control as well as the nonlinearities 
introduced by the adaptive law in the feedback control system do not allow the use of the powerful control 
design techniques developed for LTI plants. Because of these two main reasons a number of researchers 
explored the possibility of controlling plants with unknown parameters without necessarily following the 
traditional adaptive control procedure that relies on the certainty equivalence principle. This let to a wide 
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class of adaptive schemes with and without parameter estimation, with switching and supervisory logic in 
an effort to utilize as much as possible tools from robust control of known LTI systems. In the following 
sections we discuss some of these main approaches. 


35.3.1 Switching and Multiple Models 


In multiple model adaptive control (MMAC) schemes [22,31], since the plant parameters are unknown, 
the parameter space is parameterized with respect to a set of plant models, which is used to design a finite 
set of controllers, so that each plant model from the set can be stabilized by at least one controller from 
the controller set. A switching approach is then developed so that the stabilizing controller is selected 
online based on the I/O data measurements. The general structure of this MMAC with switching, as it is 
often called, is shown in Figure 35.5. 

In Figure 35.5, N controllers are used to control a plant whose parameters 0* are unknown or could 
change with time. The a priori knowledge of where the elements of 6* are located, such as lower and 
upper bounds, is used to parameterize the plant and generate a finite set of nominal models M1,..., My, 
as well as a finite set of candidate controllers C),...,Cy so that for each possible plant there exists at 
least one stabilizing controller from the set of the N controllers. This by itself could be a difficult task in 
some practical situations where the plant parameter regions are uncertain or change in an unpredictable 
manner. Furthermore, since there is an infinite number of plants within any given bound of parametric 
uncertainty, finding controllers to cover all possible parametric uncertainties may also be challenging. 
It is usually assumed that the set of controllers with the property that at least one of them is stabilizing 
is available. Once the set of controllers with the stabilizing property is available, the problem of finding 
the stabilizing one using I/O data has to be resolved. This is achieved by the use of a switching logic that 
differs in detail from one approach to another. 

An example of such logic is described as follows: each nominal model Mj is used to construct an output- 
predictor E;, a dynamical system whose inputs are the output y and the input u of the unknown plant, 
and whose output is the prediction output y;. Each j; will converge to y asymptotically if the transfer 
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FIGURE 35.5 The MMAC architecture. 
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function of the unknown plant is equal to the nominal process model transfer function M; and there 
is no noise or disturbances. Controller selection is decided by comparing in real-time suitably defined 
norm-squared prediction errors, also referred to as performance signal. Typically, the performance signal 
is an exponentially weighted integral of prediction errors 


t 
w(t) = / e ?ME-1) 6? (q) dr. (35.43) 
0 


where e;(t) = y(t) — y;(£) is the prediction error associated with the ith nominal model. 

In Figure 35.5, o(t) is a piecewise continuous switching signal that takes on values from the candidate 
controller index set. The candidate controller associated with the smallest performance signal is placed 
in the loop according to an appropriate switching logic. The stability results associated with the MMAC 
schemes state that there exists a number § such that if the unmodeled dynamics are smaller than §, then 
all the closed-loop signals remain bounded. Furthermore, switching stops after some finite time [32]. 

Following the idea of supervisory control, logic-based switching and multiple models are combined 
with conventional adaptive control [22] with the objective of improving the sometimes poor transient 
performance of conventional adaptive schemes. 

Another MMAC approach is the so-called robust MMAC (RMMAC), whose basic structure is described 
in Figure 35.6. RMMAC provides guidelines for designing both the candidate controller set and the 
supervisor [33,34]. The candidate controller set is designed by using robust control techniques such as the 
mixed-\1 synthesis, in order to account for robust stability and performance requirements. The supervisor 
identifies the nearest probabilistic model via a dynamic hypothesis testing, and uses the probabilities 
obtained to weight the output of each candidate controller. 
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Given a state-space description of the plant, where the system matrices depend upon a real-valued 
parameter vector 0 € (2, the space Q2 is partitioned into N small subsets Q2;. For each subset 2; a candidate 
controller C; is designed and a stochastic LTI model Mj is developed, including disturbance covariances, 
which are used to design a set of N Kalman filters (KFs), one for each model Mj. 

The supervisor comprises the set of KFs plus a posterior probability evaluator. For each KF, the residual 
ri(t) = y(t) — p;(t|t — 1) is calculated. The posterior probability P;(t), that is the probability that the model 
corresponding to the ith KF is the true one, is computed using the recursive formula 


Byel (OS; 'rilt) 


SN, Be OF 
J= 


P(t) = 1) (35.44) 


with Bj; = 1/./27 det(S;), and S; is the residual covariance matrix, which can be precalculated using the 
steady-state covariance equations for the KF. The control input is the sum of the output of each candidate 
controller, weighted with the probabilities obtained: 


N 
u(t) = >> Pi(t)ui(t). (35.45) 


i=1 


The RMMAC scheme requires the disturbances to be Gaussian random variables with known covari- 
ance. Furthermore, the KFs must satisfy a Baram proximity measure (BPM) requirement (see [33] for 
details). When all the assumptions are satisfied, extensive simulations show, at least empirically, the 
goodness of the method under various scenarios. A drawback of the scheme is that the performance 
of the RMMAC scheme may be sensitive to model assumptions (disturbance model, initial conditions, 
etc.). For this reason, the same authors developed a variant architecture, called RMMAC/XI, in order 
to account for possible performance degradation of the standard RMMAC when the plant disturbances 
have wide variability, at the expense of doubling the number of KFs. Another drawback of the method is 
that currently, there are no stability results. Despite these drawbacks, the scheme has been demonstrated 
to perform very well compared to other schemes in many situations. Recently, the RMMAC scheme was 
integrated with the Stability Overlay (SO), an algorithm that can be integrated with virtually any MMAC 
architecture, guaranteeing the stability of the closed-loop system, while preserving the high levels of per- 
formance observed in the simulation of the standard RMMAC, whenever the model assumptions are not 
violated [35]. The SO is based on a falsification philosophy: a controller receives a “reward” if a stability 
test is satisfied and the controller is disqualified or not, based on its rewards. However, unlike [36], where 
the rewards are also used to achieve performance, the SO is only responsible for the I/O stability of the 
plant, constraining the controllers that can be selected at each sampling time. The RMMAC algorithm is 
run in parallel in order to satisfy the posed performance requirements. 


35.3.2 Unfalsified Adaptive Control 


Another class of supervisory-based adaptive control schemes proposed in the literature aimed to relax 
the assumptions on the plant model required in the multiple model adaptive schemes. This family of 
nonidentifier-based schemes, referred to as unfalsified adaptive switching control (UASC), shown in 
Figure 35.7, compares at every time instant certain performance-related test functionals generated using 
a virtual reference input signal [36,37]. The currently operating controller is substituted whenever its 
tested performance turns out to be worse than the inferred performance of another candidate controller. 
Unfalsified control requires no a priori knowledge of the plant [26], and relies solely on I/O data to choose 
the stabilizing controller from a given set of controllers. 

Given an unknown plant P and a family of N LTI controllers C = {C;(s),i € Nj, the switching super- 
visor aims at controlling the uncertain plant by switching at any time a selected controller C,() from 
a set of candidate controllers according to a specified switching logic. The so-called fictitious reference 
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FIGURE 35.7 The UASC architecture. 


signals or virtual reference v;, are constructed using I/O data. Given at time f a set of past measured data 
(u(-), y(-)) obtained over [0, t] from the feedback system (P, C,,.)), vi equals the fictitious reference that, 
if injected into the feedback system (P, C;) would reproduce the unknown plant I/O pairs (u(-), y(-)). 

For a candidate controller C;, the performance index, or test functional 7; is calculated from past I/O 
data. The functional is a measure of performance of the closed-loop system over the interval [0, t], had 
the controller C; been in the closed loop with the corresponding v; as the reference signal and the past 
I/O pairs (u(-), y(-)) as the measured data. A typical test functional used in UASC is 


leFll5 + ell 5 


“(t)= i(t), ‘ = 
Tie) = ax, LO) Lie) = 


(35.46) 


where p > 0 is a design constant, €;(t) 4 y(t) - y(t), and m= >0 prevents the denominator from assum- 
ing values too close to zero. The notation ||x*||2 stands for the J, norm of the sequence x(-) up to time t. 
The selection of the controller index o(t) is made, at each time ft, via a hysteresis switching logic [24]. The 
virtual reference is well defined, provided that the C;’s be causal and stably causally invertible (CSCI). 
While the results summed up hinge upon the assumption that all C;’s be CSCI, in [38], it is proved that 
the same conclusions hold true for possibly nonstably invertible C;’s provided that a modified virtual 
reference, be appropriately used. 

The main positive feature of UASC schemes is the fact that, even in the absence of any prior information 
on the uncertain plant (the plant can be ofany order, unstable, nonminimum phase, nonlinear or of infinite 
order), they can select in finite time a final controller yielding, a finite affine gain from the reference to 
the data, under the minimal conceivable requirement, namely the existence of a stabilizing candidate 
controller in the candidate controller set. One of course could argue that to design a stabilizing controller 
to start with you need to know something about the plant. One major drawback is that there is no stability 
result that guarantees that convergence to the stabilizing controller will be achieved in general. In fact, 
the system could converge to an unstable controller leading to an internally unstable closed-loop system 
when the adaptive procedure is switched off. Since the method is driven by data, the quality of the data 
is crucial. If the data do not carry information about the plant or are corrupted by initial conditions 
and/or disturbances and other plant uncertainties the wrong controllers could be selected leading to 
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FIGURE 35.8 Detail of multi-model UASC. 


large transients which from the practical point of view are unacceptable. Such bad transients have been 
observed by independent simulation studies [39,40]. 

When candidates of possible plant models are available, they can be incorporated into the design 
of the candidate controllers, and the construction of test functionals, to obtain a multimodel UASC 
(MUASC) scheme with improved performance [27]. The multimodel based UASC approach consists 
of embedding in the UASC scheme a family of nominal models Mj pairwise associated with the given 
candidate controllers C; in such a way that the feedback interconnection (Mj, C;) is stable. 

Consider now, according to Figure 35.8, the feedback interconnection of each model M; with the 
corresponding C; 


| yilt) = M;(u;)(t), 
uj(t) = Ci(vj — yi)(t), 


where v; is the virtual reference input for the ith controller. Define z;(t) = [ui(t) yilt)]’, and Z;(t) = 
[ uj(t) ¥j(t) ]’ = z(t) — z(t). Note that Z; measures the discrepancy of (P, C;) from the tuned-loop (Mj, C;). 

The MUASC algorithm replaces the test functional (Equation 35.46) with the multimodel-based test 
functional 


Fit) = max Aji(t), (35.47) 
: if ||z“ll, = 0, 
1) is 7 2 
x =| le elsewhere. (35.48) 
iW2 


In [27], it is shown that the computation of v; can be avoided in order to calculate Equations 35.47 
and 35.48, so that the validity of Equations 35.47 through 35.48 holds true for general controllers C;, even 
when, as in the case of nonstably invertible controllers, v; is not well defined. 

Asin UASC, for the MUASC scheme, stability is guaranteed under the minimal conceivable assumption 
that a stabilizing candidate controller exists. Furthermore, with MUASC, guidelines are provided how 
to design the family of nominal models and the family of candidate controllers, so that the number of 
controller switching is reduced and the chance that destabilizing controllers be switched-on is moderated, 
which helps reduce transients. These properties cannot in general be guaranteed in schemes with no 
nominal models. Extensive simulation studies demonstrate the advantages of MUASC. 
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35.4 Mixed Identifier and Nonidentifier-Based Tools 
35.4.1 Adaptive Control with Mixing 


To reduce sensitivity to model assumptions that occurs in RMMAC, a design, called adaptive mixing 
control (AMC), shown in Figure 35.9, substitutes the posterior probability evaluator with a robust online 
parameter estimator plus a mixing strategy for the output of each candidate controller [41]. 

The supervisor consists of two subsystems: the online parameter estimators and the mixer. By monitor- 
ing the results of the robust parameter estimators at each time t the adaptive control mixer decides which 
outputs of controllers to combine and pass on to the plant. The candidate controllers are designed to 
handle overlapping parameter sets and therefore mixing their outputs up when the estimated parameters 
fall into these overlapping parameter sets is appropriate. Another advantage is that no switching takes 
place but rather a smooth transition from one controller or combination of controllers to another. We 
present below some of the technical details and stability results of the AMC approach. Consider the SISO 
LTI plant: 


y = Go(s, O°) + Am(s))(Uu+E) +6, (35.49) 


where Go(s, 8*) represents the nominal plant; the vector 6* € Q contains the unknown parameters of 
Go(s, 8*); Am(s) is an unknown multiplicative perturbation; and &, ¢ are bounded disturbances, that is, 
IS(t)| < 0, I¢(t)| < bo, Vt = 0. 

As with the MMAC schemes, the parameter uncertainty set Q is divided into n smaller subsets (2;, and 
a family of N candidate controllers C;(s), . .. , Cy(s) is designed using the powerful tools from robust con- 
trol for LTI plants. We use a mixing strategy to develop the multicontroller C(B, 8), which is constructed 
from C;(s),...,Cy(s) by taking into account linear combinations of these controllers. The multicon- 
troller depends on the mixing signal B ¢ RY which determines the participation level of the candidate 
controllers generating a mix of candidate control laws. For fixed values of 6 € [0, 1], the multicontroller 
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FIGURE 35.9 The AMC architecture. 


35-20 Control System Advanced Methods 


u = —C(s;, 8)y has the state-space form 


xc = Ac(B, ®)xc + Bc(B, 9)y, 
u=—Cc(8, 9)xc. 


The mixer determines the participation level B; of each candidate controller, through the map B : 2 t= 
[0,1]‘. Construction of the multicontroller involves interpolating the candidate controllers over the 
parameter overlaps. The mixing strategy must be designed to ensure stability on the overlapping regions. 
As in gain scheduling, interpolation methods may not satisfy pointwise stability over the set Q; therefore 
the controllers should be designed by taking into account the interpolation used in order to guarantee 
stability. 

The adaptive mixing law approach replaces 0* with its estimate 0. Because of the presence of multiplica- 
tive uncertainty Aj(s) and disturbances &, ¢, a robust online parameter estimator, like those developed 
in Section 35.2.6 is used. 

The main stability properties of the AMC scheme for the case of regulation are [41] 


1. If Am, &, 6 =0, then u, y > 0, u, y > Oast > ov. 
2. If Ams &¢ 40, There exists j1* > 0 such that, if cA} < w*, where A; depends on A,,, and c > 0 
is a finite constant, then the AMC scheme guarantees u, y, u, y € Loo, and 


t 
i ly(x)|? dt < cop?t+cy, (35.50) 
0 
where 1” = c(Aj + & + 4). 


The stability properties and performance in simulations of different schemes have been studied using 
a benchmark example in [40] demonstrate the advantages and disadvantages of the different approaches. 


35.5 Conclusions 


The area of robust adaptive control has grown to include classes of different techniques and algorithms 
based on different assumptions with the goal to control plants with large parametric uncertainties which 
cannot be handled by nonadaptive schemes. A significant achievement of most of these efforts is that 
robust adaptive control in general can effectively deal with parametric uncertainties as no other schemes. 
In addition, it can also handle modeling errors, bounded disturbances, noise, and so on. While some 
schemes perform better than others, in general, all schemes suffer from the same fundamental limitations. 
Since they all rely on online learning to design or choose the appropriate controller, the plant performance 
relies on the quality of learning which in turn relies on the quality of the I/O data it processes. Corruption 
of the I/O data by initial conditions, disturbances, modeling errors, and so on could influence learning 
and lead to wrong selection of controllers for some intervals of time especially initially. This will lead 
to large transients which in turn improve the situation as the signal-to-noise ratio increases aiding the 
learning process and leading to the selection of a better controller. The scheme that causes the least 
transients is obviously better. Expecting an adaptive scheme to have similar transient behavior as in 
the nonadaptive case is very unrealistic and possibly counter intuitive in general. Another fundamental 
limitation of robust adaptive control is that the closed-loop system is nonlinear. Therefore, poles and 
zeros no longer make sense and measures such as gain and phase margins, bandwidth, and so on are 
not applicable as they are only defined for LTI systems. The nonidentifier-based schemes aim to reduce 
this drawback by switching from one LTI controller to another so that at least within each time interval 
the closed loop system behaves as an LTI system. Even in this case, however, convergence to the desired 
stabilizing controller cannot be guaranteed unless there is persistence of excitation which is unrealistic in 
most practical situations. Convergence to a nonstabilizing controller which in the closed loop guarantees 
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zero regulation or tracking error is possible and this is what is usually achieved. As long as the scheme is 


switched-on, then every thing will stay as it is till the plant parameters change and that is when switching 
will start again. The library of adaptive control tools developed together with a good understanding of the 
advantages and limitations of different tools as well as a good understanding of the limitations associated 


with trying to control a plant while at the same time trying to learn its parameters from I/O data are very 
important in every application of robust adaptive control. 


References 


DW w 


K.S. Tsakalis and P.A. Ioannou. Linear Time Varying Systems: Control and Adaptation. Prentice-Hall, 
Englewood Cliffs, NJ, 1993. 

H.P. Whitaker, J. Yamron, and A. Kezer. Design of Model Reference Adaptive Control Systems for Aircraft. 
Report R-164, Instrumentation Laboratory, MIT, Cambridge, MA, 1958. 

R.E. Kalman. Design of a self-optimizing control system. Trans. ASME, 80:468-478, 1958. 

R.E. Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, 1957. 

A.A. Feldbaum. Optimal Control Systems. Academic Press, New York, 1965. 

P.C. Parks. Lyapunov redesign of model reference adaptive control systems. [EEE Trans. Automat. 
Control, 11:362-367, 1966. 

A.S. Morse. Global stability of parameter adaptive control systems. [EEE Trans Automat Control, 
25:433-439, 1980. 

KS. Narendra, Y.H. Lin, and L.S. Valavani. Stable adaptive controller design, Part II: Proof of stability. 
IEEE Trans. Automat. Control, 25:440-448, 1980. 

I.D. Landau. Adaptive Control: The Model Reference Approach. Marcel Dekker, New York, 1979. 

G.C. Goodwin, P.J. Ramadge, and P.E. Caines. Discrete-time multi-variable adaptive control. [EEE 
Trans. Automat, Control, 25:449-456, 1980. 

B. Egardt. Stability of Adaptive Controllers, Lecture Notes in Control and Information Sciences, Springer- 
Verlag, Berlin, 1979. 

P.A. Ioannou and P.V. Kokotovic. Instability analysis and improvement of robustness of adaptive 
control. Automatica, 20:583-594, 1984. 

C.E. Rohrs, L. Valavani, M. Athans, and G. Stein. Robustness of continuous-time adaptive control 
algorithms in the presence of un- modeled dynamics. IEEE Trans, Automat. Control, 30:881-889, 1985. 
P.A. Ioannou and J. Sun. Theory and design of robust direct and indirect adaptive control schemes. Int, 
[_ Control, 47:775-813, 1988. 

L. Praly. Robust model reference adaptive controllers, Part I: Stability analysis. Proc. of the 22nd IEEE 
Conference on Decision and Control, San Antonio, Texas, 1984. 

P.A. Ioannou and K.S. Tsakalis. A robust direct adaptive controller. JEEE Trans. Automat. Control, 
31:1033-1043, 1986. 

M. Krstic, I. Kanellakopoulos, and P. Kokotovic. Nonlinear and Adaptive Control Design. John Willey 
& Sons Inc, New York, 1995. 

Y. Zhang, B. Fidan, and P.A. Ioannou. Backstepping control of linear time-varying systems with known 
and unknown parameters. IEEE Trans. Automat, Control, 48:1908-1925, 2003. 

B. Fidan, Y. Zhang, and P.A. Ioannou. Adaptive control of a class of slowly time-varying systems with 
modeling uncertainties. IEEE Trans. Automat, Control, 50:915-920, 2005. 

M.M. Polycarpou. Stable adaptive neural control scheme for nonlinear systems. [EEE Trans Automat 
Control, 41:447-451, 1996. 

H. Xu and P.A. Ioannou. Robust adaptive control of linearizable nonlinear single input systems with 
guaranteed error bounds. Automatica, 40:1905-1911, 2003. 

K.S. Narendra and J. Balakrishnan. Adaptive control using multiple models. IEEE Trans. Automat. 
Control, 42:171-187, 1997. 

B.D.O. Anderson, T.S. Brinsmead, D. Liberzon, and A.S. Morse. Multiple model adaptive control with 
safe switching. Int. J. Adapt. Control Signal Process, 15:445-470, 2001. 

A.S. Morse, D.Q. Mayne, and G.C. Goodwin. Applications of hysteresis switching in parameter adaptive 
control. IEEE Trans. Automat. Control, 37:1343-1354, 1992. 

P.V. Zhivoglyadov, R.H. Middleton, and M. Fu. Localization based switching adaptive control for 
time-varying discrete-time systems. JEEE Trans. Automat. Control, 45:752-755, 2000. 

M. Stefanovic and M.G. Safonov. Safe adaptive switching control: Stability and convergence. IEEE 
Trans. Automat. Control, 53:2012-2021, 2008. 


35-22, Control System Advanced Methods 


27. S. Baldi, G. Battistelli, E. Mosca, and P. Tesi. Multi-model unfalsified adaptive switching supervisory 
control. Automatica, 46:249-259, 2010. 

28. P.A. Ioannou and J. Sun. Robust Adaptive Control. Prentice-Hall, Englewood cliffs, NJ, 1996. 

29, P.A. Ioannou and B. Fidan. Adaptive Control Tutorial. Advances in design and control, SIAM, 
Philadelphia, PA, 2006. 

30. P.A. Ioannou and A. Datta. Robust adaptive control: Design, analysis and robustness bounds, in P.V. 
Kokotovic Ed., Grainger Lectures: Foundations of Adaptive Control, Springer-Verlag, New York, 1991. 

31. J.P. Hespanha, D. Liberzon, and A.S. Morse. Hysteresis-based switching algorithms for supervisory 
control of uncertain systems. Automatica, 39:263-272, 2003. 

32. A.S. Morse. Supervisory control of families of linear set-point controllers, Part 1: Exact matching. IEEE 


Trans Automat, Control, 41:1413-1431, 1996. 


33. S. Fekri, M. Athans, and A. Pascoal. Issues, progress and new results in robust adaptive control. Int_J. 


Adapt. Control Signal Process., 20:519-579, 2006. 
34, S, Fekri, M. Athans, and A. Pascoal. Robust multiple model adaptive control (rmmac): A case study. 


Int. J. Adapt. Control Signal Process., 21:1-30, 2007. 

35. P. Rosa, J.S. Shamma, C.J. Silvestre, and M. Athans. Integration of the stability overlay (so) with 
the robust multiple-model adaptive control (rmmac). Proc. of the 17th Mediterranean Conference on 
Automation and Control, Thessaloniki, Greece, 2009. 

36. M. Stefanovic, R. Wang, A. Paul, and M.G. Safonov. Cost detectability and stability of adaptive control 
systems. Int, J. Robust Nonlinear Control, 17:549-561, 2007. 

37. M.G. Safonov and T.C. Tsao. The unfalsified control concept and learning. JEEE Trans. Automat, 
Control, 42:843-847, 1997. 

38. C. Manuelli, $.G. Cheong, E. Mosca, and M.G. Safonov. Stability of unfalsified adaptive control with 
non-scli controllers and related performance under different prior knowledge. Proc. of the European 
Control Conference, Kos, Greece, 2007. 

39. A. Dehghani, B.D.O. Anderson, and A. Lanzon. Unfalsified adaptive control: A new controller imple- 
mentation and some remarks. Proc. of the European Control Conference, Kos, Greece, 2007. 

40. S. Baldi, P. Ioannou, and E. Mosca. Evaluation of identifier based and non-identifier based adaptive 
supervisory control using a benchmark example. 4th International Symposium on Communications 
Control and Signal Processing, Limassol, Cyprus, 2010. 

41. M. Kuipers and P.A. Ioannou. Multiple model adaptive control with mixing. [IEEE Trans. Automat. 
Control, to appear in 2010. 


36 


Iterative Learning 
Control 


Bol) TittotMcnOline acca cienecinsauns 36-1 
Implementation Hardware « Outline 

86.2. Example: A Serve: Systeni sssisssasetacsosamendcanneies 36-3 

36.3 PFrequency-Domain Deésigtt....:.c.cccssccsessscssescsess 36-6 


Frequency-Domain Analysis « Tuning Based on 
Nyquist Analysis » Zero-Phase Q-Filter « Hoo 


Methods 
36.4 Generalized Time-Domain System 
ICSCr PRON csirscciaeurincia anemia 36-12 
Douglas A. Bristow 36.5 Time-Domain Analysis .....:ccsesesesssssoesersesses 36-13 
Missouri University of Science and Technology Asymptotic Stability « Monotonic Convergence 
: Trial-Varying Disturbances « Robustness 
Kira L. Barton : ee . : 
ieee eae : 36.6 Time-Domain Norm-Optimal Design.......... 36-15 
University of Illinois at Urbana-Champaign 1 oad ; A 
Weighting Matrix Design 
Andrew G. Alleyne 36.7 Concluding Remarks... : 
University of Illinois at Urbana-Champaign Rete SGC S .sissinstaasniscigaisananemisiiaisreenasinananneits 


36.1 Introduction 


Iterative learning control (ILC) is a performance-enhancing feedforward control scheme for systems that 
repeat the same trajectory or task. Before the start of each iteration of the trajectory, the designed ILC 
algorithm uses the error signal from the previous iteration(s) to generate an updated feedforward control 
signal. The learning process converges after anywhere from a few to tens of iterations, depending on 
the algorithm. In the literature it is commonly reported that ILC improves the performance of physical 
systems by several orders of magnitude, measured by root mean square (RMS) or maximum error, as 
compared to those systems’ feedback controllers. 

The essential caveat in ILC is system repeatability, meaning that multiple repetitions of the trajectory 
or task yield nearly identical error signals. Although this may appear to be very restrictive, many practical 
systems are highly repetitive. This is particularly true in manufacturing systems, as will be discussed 
shortly. When the system is repeatable, one expects that for a given trajectory there exists a control signal 
that yields zero or nearly zero error. In practice, it is rare that this control signal is known a priori. 
However, in applications where the trajectory repeats ILC is used to search for, or learn, the optimal 
control signal. The process is illustrated in Figure 36.1. 

ILC was originally developed in the mid-1980s for robotics [2] and, although robotics remains an 
important application area, it has expanded to a much broader class of manufacturing and chemical 
processing systems. Examples (see references in [3,5,6]) include computer numerically controlled (CNC) 
machine tools, wafer stage motion systems, injection molding machines, aluminum extruders, cold rolling 
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FIGURE 36.1 Time- and iteration-domain evolution of signals in ILC. 


mills, induction motors, chain conveyor systems, rapid thermal processing, and semibatch chemical 
reactors. There are many books and surveys on ILC [3-6,9,12,13,18]. This chapter focuses on summarizing 
the main ideas in ILC and presenting established design techniques for linear systems. The reference list 
to this chapter is a good starting place for the reader interested in continuing beyond the information 
contained in the following sections. 


36.1.1 Implementation Hardware 


Before delving into the algorithms of ILC, the practicing engineer may be interested in the hardware 
needs of such an advanced control algorithm. Most advantageously, ILC is usually implemented as an 
offline algorithm, in between iterations. In an offline mode, the ILC algorithm can usually be computed 
using the control system’s existing digital signal processor (DSP) or on another computational platform 
such as a PC. In high-throughput manufacturing where offline time is expensive, a simple ILC algorithm 
is preferable to minimize downtime. Using a modern DSP, offline time should be on the order of mil- 
liseconds for all but the most complex algorithms or longest trajectories. Additionally, offline time can be 
eliminated by “freezing” the control signal after convergence. That is, learning is used only on the first few 
iterations of the process until the control signal converges. The remaining process runs skip the learning 
step, eliminating downtime, and simply reuse the already converged feedforward control signal. 

Because ILC requires the storage of error signals and control signals for offline computation, memory 
capacity (usually RAM) must be considered. The amount of memory required is calculated as follows: 


. No. of time Bits of Order of 
Memory (bits) = 2 x . x ._ X No. of axes x ; ‘ 
samples/iteration —_ resolution algorithm 


where “2” is used to account for the error signal and the control signal. 


36.1.2 Outline 


The remainder of this chapter is organized as follows. A basic ILC algorithm is introduced and tuned for 
a servo system example problem in Section 36.2. Two design frameworks are presented in this chapter. 
The first one, frequency-domain design, is presented in Section 36.3. This is a simpler approach that 
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is less general than the second one. The second technique, time-domain norm-optimal design, requires 
additional preliminary setup and results before it can be presented. As such, Section 36.4 presents the 
class of systems and Section 36.5 presents some analytical convergence and robustness results. The norm- 
optimal design is presented in Section 36.6. A brief discussion of advanced algorithms and concluding 
remarks are given in Section 36.7. 


36.2 Example: A Servo System 


Consider the discrete-time system, 


z—0.5 


Gis (z — 1)(z — 0.925)’ 


(36.1) 
whose Bode plot is shown in Figure 36.2. Discrete-time is assumed because ILC requires the storage 
of signals, which is typically done digitally. Here, G represents a servo-positioning system with viscous 
friction. Assume that the system is stabilized with a proportional feedback controller, 


C(z) = 0.425. (36.2) 
The desired output trajectory is the triangle wave (shown in Figure 36.3a), 


k/200,  0<k<200, 
yak) = / (36.3) 
2—k/200, 201 <k < 400, 


where k = 0,1,...,400 is the time index. The triangle wave is used, for example, in scanning operations 
where the forward and backward motions are scanning at the same velocity. Figure 36.3b shows the error 
signal from the feedback controller. The goal is to reduce the tracking error by adding an ILC to the 
control system. 

ILC is a plug-in-type controller that can be added to an existing control loop, as shown in Figure 36.4, 
by adding the ILC signal to the feedback controller’s signal. The reader may note that the ILC signal 
is injected in the same location in the loop that is often used for feedforward control in servo systems. 
This does not, however, exclude the use of ILC on systems that already employ a feedforward controller 
[5]. Alternatively, ILC can be added to the reference signal, which is most commonly used on closed- 
architecture control systems [5,11]. 
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FIGURE 36.2 Bode plot of the open-loop servo plant in Equation 36.1. 
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FIGURE 36.3 (a) Triangle reference signal. (b) Error signal using feedback controller, e9(k). 


For ILC design, several assumptions are made: 


Al. The reference signal repeats. Each repetition is referred to as a trial or iteration, and all iterations 
have the same length. 

A2. Any external disturbance repeats identically on each iteration. 

A3. Initial conditions (ICs) on each iteration are identical. 


Assumptions A1l-A3 are standard in ILC analysis and design. Although A2 and A3 are rarely strictly 
true, in practice iteration-to-iteration variations are often negligible in precision servo systems. The topic 
of iteration-to-iteration variation is discussed in more detail in Section 36.5. 

Possibly the simplest approach is a first-order P-type ILC algorithm as given by 


uj4i(k) = uj(k) + yej(k + 1), (36.4) 


where y is the learning gain. One can think of Equation 36.4 as an integrator in the iteration domain, as 
follows. On each iteration that the error is nonzero, the control signal grows until the error finally reaches 
zero. The reader may note that the error signal is forward shifted by one time step in Equation 36.4. This 


Iterative Learning Control 36-5 


(~ Offline computation 
(eg. networked PC) 


vo 

S| 

&< ILC algorithm 
eo) 


Memory 


> G(z) > 


Real-time Plant 
control (eg. robot) 
(eg. DSP 


FIGURE 36.4 ILC added to a feedback control loop as a “plug-in” controller. 


is standard in ILC and used to compensate for the usual one-step delay (or equivalently relative degree of 
one) in the plant, Equation 36.1. That is, uj(k) does not affect e;(k) because of the plant’s one-step delay; 
hence, e;(k+1) is used instead. 

The algorithm parameter y can be tuned to achieve convergence with satisfactory performance. One 
suitable choice, obtained through trial-and-error tuning, for the previous example is y = 0.1. The RMS 
error iteration series for this choice is shown in Figure 36.5. As shown in the figure, the error is asymp- 
totically approaching zero at a rate of approximately one order of magnitude for every 50 iterations. 
Figure 36.6 shows the error, feedback control, and ILC time series for iterations 0, 1, 10, and 100. As the 
ILC converges, the control effort transfers from the feedback controller to the ILC, as shown by the signals 
in Figure 36.6. One measure of effectiveness of a feedforward control, such as ILC, is the magnitude of 
feedback control effort. 

The P-type algorithm is the simplest algorithm that can be used. As a cautionary note, the P-type 
algorithm may not be sufficient for many systems. In particular, high-frequency resonances, common in 
servo systems, may be problematic for P-type learning [11]. In practice, it is more common to use an 
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FIGURE 36.5 RMS error iteration series using P-type ILC. 
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algorithm of the form 
uj+i(k) = Q(z)(uj(k) + L(z)e(k)), (36.5) 


where the mixture of time-domain signals and frequency-domain filters is a (standard) abuse of nota- 
tion. The Q-filter is a low-pass filter used to limit the learning bandwidth to provide robustness and is 
particularly useful in servo systems with lightly damped system resonances. The L-filter, also called the 
learning filter, is designed to maximize the learnable bandwidth and convergence rate. Q and L designs 
are discussed in more detail in the following sections. 


36.3 Frequency-Domain Design 


The frequency-domain design framework uses classical control design tools such as z-transform and 
Nyquist plots [11,14]. A key additional assumption in this framework is that the iteration length (or 
reference length) is infinitely long. This is, of course, an approximation of reality because the iteration 
length is always finite in practice. However, this assumption permits the use of the z-transform, which 
greatly simplifies the design. 
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Assuming zero ICs, the output of the system shown in Figure 36.4 on the jth iteration can be calculated as 


_ _C(z)G(z) G(z) 
~ 14+C@)G@) Yaz) Tr c@ew Ui: (36.6) 


Yj(z) 


The error on the jth iteration is Ej(z) = Yq(z) — Y;(z), or, 


Ej(z) = Eo(z) — H(z) U,(z), (36.7) 
where 
= G(z) 


is the transfer function from the ILC input to the error and 


1 
Eo(z) = Tac@ow 4? (36.9) 


is the initial error, or the error from the feedback control. It is easy to show that the addition of nonzero 
ICs and exogenous disturbance signals appear as additional terms in Eo(z). Furthermore, changing the 
location of where the ILC enters into the control loop, such as the reference signal [5,11], changes only 
H(z). Thus, one can conclude that the form of Equation 36.7 is general, although Eo(z) and H(z) depend 
on the specific control system architecture and disturbances. 


36.3.1 Frequency-Domain Analysis 


The frequency-domain description of the first-order ILC algorithm (Equation 36.5) is 
Ujr(2) = Az)(Uj(z) + LEZ). (36.10) 
Substituting Equation 36.7 into Equation 36.10, the iteration dynamics can be obtained as 
Uj+1(Z) = Q(z) — L(z)H(z)) Uj(Z) + Q(z) L(z)Eo (2). (36.11) 


The goal in designing L(z) and Q(z) is to force a contractive mapping from Uj;(z) to Uj+1(z), which is 
done by making 


| Q(z) — L(z)H(Z))lloo <1, (36.12) 


where |/e(Z)||oo = Max_g<w<n |e(e)|. The contractive mapping ensures convergence of U;(z), which 
can be calculated from Equation 36.11 as 


ites, QAz)L(2) 
Uoo(Z) = Uj(z) = 1 a@d— L(@@H@) (36.13) 


Substituting Equation 36.13 into Equation 36.7 yields the converged tracking error as 


Eoo(Z) = Eo(z) — H(z)Uo0(z) = ( = aan ia) Eo(z). (36.14) 
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36.3.1.1 The Role of the Q-Filter 


Fundamental to frequency-domain design is the low-pass Q-filter, which ideally has the frequency-domain 
characteristic 


Qideal (ce) = 


(* QSOS ae (36.15) 


0, wo <o<t 


where ), is the cutoff frequency. For Qideat(e), the stability condition (Equation 36.12) simplifies to 
[1 —L(e)H(e)| <1 for0<@< waz, (36.16) 


and the asymptotic error is given by 


, , 0, 0<ao<aw 
ied Cag es an ee ae (36.17) 
Eo(e), Oe <@<T. 


Thus, the low-pass Q-filter simplifies the design problem by limiting the frequency range over which 
Equation 36.16 must be satisfied, but at the expense of performance. In fact, in the ideal Q-filter analysis, 
Q acts as a learning switch. When Q is unity at a particular frequency, that frequency is turned on 
for learning. When Q is zero at a particular frequency, that frequency is turned off. The goal in the 
Nyquist tuning approach presented in the following subsection is to design L to maximize the range 
0 < w < w, over which Equation 36.16 is satisfied. In doing so, Q can be “turned on” for the largest range 
of frequencies. 


36.3.2 Tuning Based on Nyquist Analysis 


The Nyquist-based tuning method, referred to more commonly in the literature simply as frequency- 
domain design [11], is a two-step design process as illustrated in Figure 36.7. The first step is to select a 
filter type for L(z) and then to tune the filter parameters so that the Nyquist plot of L(e)H(e!®) fits inside 
the unit circle centered at one for as high a frequency range 0 < w < w, as the designer is able to achieve. 
This is equivalent to satisfying Equation 36.16 over the same frequency range. The second step is to 
construct a low-pass Q(z) (e.g., Butterworth, Chebyshev Type I and II, etc.) that satisfies Equation 36.12. 
For an ideal low-pass filter, the bandwidth is w,, although for practical filters the actual bandwidth may 
vary. The result will give near-zero tracking error up to w, frequency. 

Any type of filter may be used for L(z), although some popular choices are listed in Table36.1. The 
phase lead (and PD-type to a lesser extent) adds positive phase to partially compensate for the plant’s 
phase lag. The model inversion type is intended to fully compensate for the plant’s dynamics, but may 
be challenging to implement in practice. First, the inverse of the plant may not be stable, although stable, 
approximate inversion [17] can be used. Second, because many plants have high-frequency rolloff, the 
inverse results in high-frequency gain, which may amplify noise. Third, the designer may be unaware of 
the model’s inaccuracy and the inverse may create a false perception of attainable bandwidth. However, 
this may be true of any filter choice and the designer should take appropriate care in implementation and 
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FIGURE 36.7 Nyquist tuning two-step design procedure. 
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TABLE 36.1 Common Filter Types for L(z) 


Filter Name Equation 

P-type L(z) = yz 

PD-type L(z) = yz+ (z-1) 
Phase lead L(z) = yzs 

Model inversion L(z) ® H71(z) 


testing. When possible, it is safest to “tune up” the Q-filter bandwidth online by starting low and slowly 
increasing w, through repeated experiments. 


36.3.2.1 Servo System Revisited: Robustness 


Figure 36.8 shows the Nyquist plots of H(z) and L(z)H(z) for the servo system ILC design presented in 
Section 36.2. Recall that for this example, L(z)=0.1z. The 0.1 gain reduces the magnitude of H(z) and the 
z adds enough phase lead to rotate the high frequencies of H(z) into the unit circle centered at one. To 
illustrate robust design, consider now a high-frequency perturbation to H(z), 


y 
ak Sera TT 

The Nyquist plots for the original system L(z)H(z) and the perturbed system L(z)H’(z) using the same 
learning filter are shown in Figure 36.9. The perturbed system leaves the unit circle at w, = 0.55 rad/sample 
and therefore is not stable with the same learning filter as the nominal system. The designer has two options 
to robustly stabilize the nominal and perturbed system. The designer can look for a learning filter that 
stabilizes both systems, or the designer can use a Q-filter to limit the learning bandwidth to the frequency 
range that is stable for both systems, w € [0, 0.55]. The latter option is explored next. 

The stabilizing effect of the Q-filter can also be seen using other frequency-domain tools with which 
the user is comfortable. Here we use a Bode plot to demonstrate that the same tools from classical control 
apply. In Figure 36.10, a Bode plot of Q(z)(1 — L(z)H(z)) is shown for several scenarios. The goal is to 
keep the magnitude below one for all frequencies, which equates to stability (Equation 36.12). The solid 
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FIGURE 36.8 Nyquist plot of the servo system example. 
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FIGURE 36.9 Nyquist plot of the perturbed servo system. 


line shows the unperturbed plant and the P-type learning function (Equation 36.4) presented in Section 
36.2 (in this case, Q(z) = 1). As noted previously, this learning algorithm is stable, which is confirmed on 
the Bode plot with magnitude less than 0 dB. The dashed line shows the Bode plot for the perturbed plant, 
H'(z). In this case, the magnitude crosses 0 dB at w = 0.55 rad/sample in agreement with the Nyquist 
plot of Figure 36.9. The dash-dot line shows the perturbed plant again, but this time with a second-order 
Butterworth Q-filter with cutoff frequency w, = 0.55 rad/sample. The Butterworth Q-filter successfully 
reduces the magnitude below 0 dB to achieve a stable learning algorithm. 

As discussed previously, the Q-filter provides robust stability at the cost of asymptotic performance. 
The performance loss can be minimized through careful selection of the cutoff frequency so as not to be 
overly conservative. Filter type and order will also affect performance. Generally, sharp filter “rolloff” is 
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FIGURE 36.10 Bode plot of the perturbed servo system with second-order Butterworth Q-filter. 
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preferred to slow “rolloff’ because it better approximates the ideal low-pass filter. Zero-phase Q-filters, 
or filters without lag, are also preferable (and popular) in ILC. These filters are discussed in the following 
subsection. 


36.3.3 Zero-Phase Q-Filter 


Phase lag in the Q-filter adds delay to the control signal, which degrades the asymptotic performance, 
particularly for high acceleration motions such as steps. The phase lag can be eliminated by using a 
zero-phase Q-filter. The zero-phase Q-filter is noncausal, although this is not problematic in ILC because 
the filtering is done offline after the entire signal has been collected. There are a number of methods for 
creating a zero-phase filter. Two methods, an infinite-impulse response (IIR) method and a finite-impulse 
response (FIR) method, are developed below using basic signal processing techniques. 


36.3.3.1 Filtfilt (IIR Method) 


This approach uses any nonzero-phase low-pass filter F(z) in a modified filtering procedure to produce 
zero-phase shift. The procedure is as follows. 


1. Filter the input signal; s;(k) = F(z)sin(k), where sin(k) is the signal to be filtered. 
2. Reverse the resultant signal in time; s2(k) = s,(N — k). 

3. Filter the signal again; s3(k) = F(z)s2(k). 

4. Reverse the resultant signal in time; sout(k) = s3(N — k). 


The phase lag introduced in step 1 is canceled in step 3 resulting in zero total phase shift. Note that the 
effective gain of the process is |F(e)|?, owing to the double-filtering. 


36.3.3.2 FIR Convolution (FIR Method) 


In this approach, a zero-phase filter is constructed using an FIR approximation of the filter. Let f(k), 
k=0,1,..., Nerr — 1 be the Ngyr-step truncation of the impulse response of F(z). Then, 
h(k) 
irc) = =>? 
ee hie hQ) 
where h(k) = f (k) * f(—k) with*defined as the convolution symbol is an FIR moving average filter with 
zero phase. 


36.3.4 Ho. Methods 


The Hoo method [7] is the reverse of the Nyquist tuning method in that Q(z) is selected first, and L(z) is 
designed second. Q(z) can be a low-pass filter of any type, and L(z) is optimally obtained as the solution 
to the model matching problem 


L*(z) = arg min ||Q(z)(I — L(z)H(z))lloo. 
L 


This can be written equivalently as a lower linear fractional transform 
Q(z)(I — L(z)H(z)) = Gur (z) + Gia(2)L(2)(I — Gr2(z)L(z)) "Gai 2), 
where 


G(z) = eae 5 


Gai(z) Gralz) 
In this form, standard Hoo synthesis tools can be used to solve the problem. If no solution is found, the 


bandwidth of Q(z) is lowered. If a solution is found, the bandwidth can be increased and a new L(z) 
obtained. 


Q(z) Q(z) 
—H(z) 0 |} 
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While the frequency domain provides a simple design approach, the next three sections present a more 
rigorous and general design technique using a time-domain system description. 


36.4 Generalized Time-Domain System Description 


Consider the discrete-time (DT), linear time-varying (LTV) system 
i(k + 1) = A(k)xj(k) + B(k)uj(k), 
ilk) = Clk)xj(k) + d(k), 
where x € R", u € R’ is the control input, y € R’ is the output, d € R’ is an iteration-invariant exogenous 


signal, k = 0,1,..., N is the time index, and j = 1,2,... is the iteration index. If xj(0) = xo for all j, the 
input-output solution for (36.18) is given by 


(36.18) 


k-1 
yi(k) = C(k) ®(k, 0)xo + se A(k, luj(l) + d(k), (36.19) 
1=0 
where 
H(k, 1) = C(k) ®(k,1+ 1)B(), (36.20) 
@(k,1l) = A(k—1)A(k—2)...A(]), ®(k,k) =I. (36.21) 


Let yq(k) be the desired output for k = 1,...,N and define the error as 


ej(k) = yak) — yj(k). (36.22) 
Then, 
k-1 
ej(k) = eo(k) — )> H(k, Duj(I), (36.23) 
1=0 
where 
ey(k) = yq(k) — C(k) ®(k, 0)xo — d(k). (36.24) 


The reader should note that (36.23) is the time-domain equivalent to (36.7). 
The lifted-system (or supervector) representation is given by 


ej = eo — Hu;, (36.25) 
where 

g=(Fa) Ff ... a(n)) , (36.26) 
uj = [u7(0) OD wre —a), (36.27) 

H(1,0) 0 0 ils 0 

H(2,0) H(2,1) 0 0 
H=]/ H(3,0) H(3,1) H(3,2) ”.- (36.28) 

: 0 

H(N,0) H(N,1) H(N,2) --- H(N,N-1) 


The lifted-system representation transforms the r-input, r-output, two-dimensional (time and 
iteration) system into an Nr-input, Nr-output, one-dimensional (iteration) system. 
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The most general first-order linear ILC algorithm for (36.25) is 
uj+1= Luu; + Lee;, (36.29) 
where Ly, Le € RN’*4". Note that (36.29) has a slightly different form than (36.5). The convention in the 


norm-optimal design presented in Section 36.6 is to use the more general Ly, Le form, while the convention 
in frequency-domain design is Q(z), L(z). For the purpose of exposition, (36.29) can be rewritten as 


ujp1 = Lay + Lee; = Q(uj + Lej), wherely=Q, Le = QL. (36.30) 
Although the lifted representation is very useful in design and analysis, it is not necessary, or oftentimes 
desirable, to use the lifted representation in the implementation because Ly and Le are large matrices when 


the iteration length is long. For example, consider the choice Ly = I and Le = diag {y(0),..., y(N — 1)}, 
y € RR’. This is the P-type algorithm, written in time domain as 


uj+1(k) = uj(k) + yk)ej(k + 1), (36.31) 
and is the discrete-time analog to the P-type algorithm for continuous time presented in Section 36.2. 


Although both are equivalent, it is much more efficient to apply the filter description (36.31) in practice 
than the matrix description (36.29) for this choice of Ly and Le. 


36.5 Time-Domain Analysis 


36.5.1 Asymptotic Stability 
Substituting (36.25) into (36.29) yields the closed-loop iteration domain dynamics 
U1 = (Lu — L.H)u; + Lee. (36.32) 


The iterative learning controller is asymptotic stability (AS) if and only if |\;(Ly —L.H)| <1 for 
i=1,2,...,Nr, where ,(e) is the ith eigenvalue of (e). Using the P-type algorithm (36.31), stability 
simplifies to |X;(I — diag{y(0), ..., y(~N — 1)}H)| < 1, or the following results: 


[1 —y(k)C(kK+1)B(k)| <1, for k=0,1,...,N. (36.33) 


This is the discrete-time analog to Arimoto et al.’s [2] original result for continuous-time systems. 
If the controller is AS, and (I — Ly + LeH) is nonsingular, then the asymptotic control can be found as 


lim uj = Uso = (I— Ly + LeH) Lee. (36.34) 


jJrw 
The asymptotic error is 


lim ej = €o0 = e9 — Huo, 
I? oe 


= [I—H(I—Ly+LeH)'Le] eo. (36.35) 
Interestingly, €4, = 0 for an AS P-type control. The ease by which perfect tracking can be achieved 


with the simplest type of iterative algorithm may be surprising. The practicality of this result, however, is 
limited in actual practice by two challenges: convergence behavior and trial-varying disturbances. 
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36.5.2 Monotonic Convergence 


For most practical systems, AS is not a strong enough condition. ILC systems that are AS may experience 
large transient learning growth, a problem studied extensively by Longman [11]. The transient learning 
growth appears as a rapid growth in the error over many iterations before convergence. Growth may 
be many orders of magnitude and convergence very slow [11], well outside the realm of practicality for 
physical implementations. 

Monotonic convergence is a stronger stability requirement than AS and is not susceptible to transient 
learning growth. Monotonic convergence of the error requires that ||eo. — ej|| decreases at each iteration, 
where |le|| is a suitable norm. For ILC design with monotonic convergence, the most commonly used 
norm is the induced 2-norm. Monotonic convergence of the control, , may also be used. (36.29) 


is monotonically convergent if 


Uso — uj| 


1 = ||(Lu — LeH)|| < 1. (36.36) 


Furthermore, | Ugo — Uj+1 | <y | Ugo — Uj | gives the rate of convergence. 


36.5.3 Trial-Varying Disturbances 


One of the key assumptions in most ILC literature is that disturbance signals repeat identically from one 
iteration to the next. In practice this is rarely the case. As will be shown, trial-varying disturbances do 
not affect stability. They do, however, affect the performance. Because ILC is a feedforward technique, 
it cannot compensate for random trial-varying disturbances. In fact, ILC will tend to learn from these 
disturbances and incorrectly attempt to compensate for them on subsequent iterations, amplifying their 
affect. Design parameters in Section 36.6 can be selected to minimize the amplification. 

Trial-varying disturbances can be treated by adding the signal 8; to the system description 


ej = eo — Hu; = 3j. 
Substituting into (36.29) yields the iteration domain closed loop 
U1 = (Lu — L.H)u; +Leeo — 1,8). (36.37) 


If 8; is bounded, then the AS from Section 36.5.1 becomes the bounded-input, bounded-output stability 
condition for trial-varying disturbances. Note that $; is filtered by Le in (36.37), and thus one expects 
trial-varying disturbance sensitivity to decrease when the gain Le is reduced. 


36.5.3.1 IC Variation 


IC variation is a problem in many systems because feedback control system limitations and nonlinear- 
ities such as friction make perfect resetting challenging. IC variation can be treated as a trial-varying 
disturbance, by setting 8;(k) = C(k) ®(k, 0)xo ;, where xo, is the IC on the jth iteration. 


36.5.4 Robustness 


As demonstrated in Section 36.3, small perturbations to the system dynamics may destabilize an ILC. 
Analytical results depend on the type of model uncertainty considered, which is an active area of research. 
In the following section, analysis is presented demonstrating that the norm-optimal design is robustly 
monotonically convergent to additive system perturbations A of the form, H; = H+ A. The largest 
system perturbation o(A) for which the ILC is robustly monotonically convergent depends on the choice 
of design parameters. 


Iterative Learning Control 36-15 


36.6 Time-Domain Norm-Optimal Design 


The time-domain or norm-optimal algorithm is designed to minimize the quadratic optimization problem 
[1,8,10], 


i: 
J= +1 QG41 + uj, Sujp1 + (uj+i = uj) R (uj4i ce uj) , (36.38) 
where {Q,S,R} are symmetric positive (semi)-definite real-valued matrices of appropriate dimension 
and H'QH +S is positive definite. Often {Q, S, R} = {qI, sI, rI} with q, s, r real-valued positive scalars. 


Applying the substitution e;,; = ej — H(uj+1 — uj), differentiating 7 with respect to uj+1, setting the 
result equal to zero, and rearranging the solution yields the norm-optimal ILC controller 


uj41 = Luu; + Lee;, (36.39) 
Ly = (H'QH+$+R)7!(H'QH +R), (36.40) 
Le = (H'QH+$+R)7'H'Q. (36.41) 


As an essential part of the design process, we discuss here some guidelines [16] for designing the 
weighting matrices by studying the properties of the ILC system with respect to convergence, performance, 
robust convergence, and performance in the presence of trial-varying disturbances. 


36.6.1 Weighting Matrix Design 


It is straightforward to verify that the norm-optimal controller is monotonically convergent in the 
control, under the 2-norm, by substituting (36.39) into (36.36). The convergence rate is calculated as 
n=l (H’QH +$+ R) R||i2. Note that the iteration-domain convergence speed depends strongly on 
R: for ||R||;. = 0 deadbeat control is achieved, as ||R||;. > oo the convergence speed approaches zero. 

The asymptotic error for norm-optimal ILC, obtained by substitution of (36.39) into (36.35), is 
C55 (1 —H (H’QH + sy HQ) eo. For zero error, ||S||;2 = 0 is necessary, and hence H'QH must be 
positive definite. An important result is that e.. is not a function of R, and hence performance is not a 
function of convergence speed. 

For robust convergence, the true system Ht = H+ A, with additive uncertainty A, is considered. Asa 
result, the requirement for robust convergence is 


[Lu — LeH lig <1 —> max (H7QH+S+R)1(R+H'Q(H— H,))| yeh, (36.42) 
1 


One can deduce that increasing ||S|| ;. allows for more robustness to model uncertainty. 

Finally, for trial-varying disturbances §;, which affect steady-state error fluctuations, one should 
minimize ||Lel|;.. From (36.39), ||Lell;2 is affected by S and R. Given that R does not affect the per- 
formance, it is the natural candidate for reducing trial-varying disturbance sensitivity. 

Based on the above information, the following tuning guidelines [16] for norm-optimal ILC control 
can be given. These guidelines are most easily implemented using common {ql sl, ri} diagonal-type 
real-valued scalar gains. 


1. Design Q to correspond to the desired weighting of the error. Usually, Q = I for uniform weighting 
of the error. 

2. The actual system dynamics will not usually be perfectly captured by the system model. Thus, 
S must be designed such that the system is robustly monotonically convergent. Start with an S 
yielding ||S||;. ~ 0.01 ||H||;. Note that the critical design parameter is the size of ||S||;. relative to 
the size of ||H||;2, where the magnitude of ||H||;. is related to system uncertainty. Subsequently, 


min 


reduce ||§||;. until the system diverges. Set ||S||;. = 2 - ||$||5™ to allow for a safety factor of 2. 
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3. When trial-varying disturbances are present, steady-state error fluctuations will occur. Start with 
||RIl;2 = 0 and increase ||R||;. until the fluctuations are within desired bounds, or the RMS error 
does not decrease anymore. 


36.6.1.1 Servo System Revisited: Norm-Optimal Design 


To illustrate the effects of varying {Q,S, R}, this subsection provides performance results for various 
combinations of the weighting matrices, where {Q = qLS=sI,R= rI}. The different weighting matrices 
are applied to the servo system example in Section 36.2. 

Figure 36.11 shows the error, feedback control signal, and feedforward control signal for {Q, S$, R} = 
{1, L I}. Similar to the results shown in Figure 36.6 the feedback signal is slowly replaced by the ILC 
feedforward signal with increasing iteration. 

To determine the effect of q on the error, s and r were held constant at {s, r} = {1, 1}, while the value of q 
was varied as {1, 0.5, 0.1, 0.001}. Figure 36.12 illustrates that the smaller the value of q, the larger the value 
of the RMS converged tracking error. Varying s affects both the performance and convergence. While 
a smaller s results in smaller converged RMS error, robustness to model uncertainty requires a larger s 
value. Holding {q; r} constant at {1,1}, respectively, Figure 36.13 illustrates how decreasing s results in a 
decrease in the RMS-converged error. 
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FIGURE 36.11 Error, feedback control, and ILC time series using norm-optimal ILC. 
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FIGURE 36.12 RMS converged error values for various Q weighting matrices. 


Lastly, r can be increased to minimize trial-varying disturbances, at the expense of convergence rate. 
Holding {4 s} constant at {1,1}, respectively, Figure 36.14 shows how the convergence rate increases as 
r increases. Figure 36.15 illustrates the effects on the RMS error of adding white noise {(mean,var) = 
(0,le-6)} to the system. As r increases, the effect of the noise on the system is decreased, as demonstrated 
by a decrease in converged RMS error. 
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FIGURE 36.13 RMS error and monotonic convergence values for varying S weighting matrices. 
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FIGURE 36.14 RMS error values for varying R weighting matrices. Note the longer iteration range. 
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FIGURE 36.15 RMS error values for varying R weighting matrices with trial-varying noise. Note the longer iteration 


range. 


Iterative Learning Control 36-19 


36.7 Concluding Remarks 


This chapter has introduced the basic ILC design and analysis tools for linear systems. The results and 
design details presented here only touch the surface of the field of ILC. In particular, a number of advanced 
algorithms have been developed, which may be of interest to the reader. Current-iteration-tracking-error 
(CITE) [6] ILC algorithms use the current iteration’s error signal in the learning algorithm, effectively 
adding feedback control in ILC design. Most notably, CITE algorithms can use the feedback component 
to compensate for trial-varying disturbances. For stochastic measurement noise, one might consider the 
use of optimal trial-varying algorithms [15]. For faster convergence rate and better robustness, high-order 
algorithms [6] use several iterations of past control and error signals in the update algorithm. 
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37.1 Input-Output Feedback Linearization 


Consider a nonlinear single-input single-output system, described by equations of the form 


x=f(x)+ g(x)u 
y = h(x) 


(37.1) 


and suppose x = 0 is an equilibrium of the vector field f(x); that is, f(0) = 0, and h(0) = 0. Assume also 
that this system has relative degree r < n at x = 0. Then there is a neighborhood U of x = 0 in R” and 
a local change of coordinates z = ®(x) defined on U [and satisfying ®(0) = 0] such that, in the new 
coordinates, the system is described by equations of the form (see Chapter 46 for details) 


Z\ = 22 
= 23 
Zp-1 = 2 
Z=vV 


(37,2) 


Zr41 = O(z) +. a(z)u 


Zr1= Qrt+1 (z) 


Zn = qu(z) 


yz. 


37-1 
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Equation 37.2, which describes the system in the new coordinates, can be more conveniently repre- 
sented as follows. Set 


Z] Zr+1 
Zz Zea 
e=(7 ] n=|7"7 J, 
Zp Zn 
and recall that, in particular, 
h(x) 
Leh(x) 
e=| 7 
r-1 
Ly h(x) 
Moreover, define 
0 1 0 0 0 0 
0 0 1 0 0 0 
AS ee we ano we eg Bet le [, 
0 0 0 -:--- 0 1 0 
0 0 0 -:--- 0 0 1 
GS (10 0-252 Or 20) 
and set 
Gr+i(Z) 
r+2(Z 
giz) = |r? 
n(Z) 


Then, Equation 37.2 reduces to equations of the form 


§ = Ag + B(bE, n) + a8, nu) 
n= 460) (37.3) 
y=Cé. 

Suppose now the input u to the system of Equation 37.3 is chosen as 


pearl 
a(n) 


This feedback law yields a closed-loop system that is described by equations of the form 


(—b(,n) +). (37.4) 


& = AE+Bv 


37.5 
i= 4(&, 0). noe 


This system clearly appears decomposed into a linear subsystem, of dimension r, which is the only one 
responsible for the input-output behavior, and a possibly nonlinear subsystem, of dimension n — r, whose 
behavior does not affect the output. In other words, this feedback law has changed the original system so 
as to obtain a new system whose input-output behavior coincides with that of a linear (controllable and 
observable) system of dimension r having transfer function 


H(s) = a ; 
st 
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Remark 37.1 


To interpret the role played by the feedback law of Equation 37.4, it is instructive to examine the effect 
produced by a feedback of this kind on a linear system. In this case, the system of Equations 37.3 is 
modeled by equations of the form 


& = AE + B(RE + Syn+ Ku) 
n= PE+ Qn 
yHcsg 


in which R and S are row vectors, of suitable dimensions, of real numbers; K is a nonzero real number; 
and P and Q are matrices, of suitable dimensions, of real numbers. The feedback of Equation 37.4 is a 


feedback of the form 
R S 1 
WS TE et ey: (37.6) 
A feedback of this type indeed modifies the eigenvalues of the system on which it is imposed. Since, 
from the previous analysis, it is known that the transfer function of the resulting closed-loop system has 
no zeros and r poles at s = 0, it can be concluded that the effect of the feedback of Equation 37.6 is such 
as to place r eigenvalues at s = 0 and the remaining n — r eigenvalues exactly where the n —r zeros of 
the transfer function of the open-loop system are located. The corresponding closed-loop system, having 
n—r eigenvalues coinciding with its n — r zeros, is unobservable and its minimal realization has a transfer 
function that has no zeros and r poles at s = 0. 


37.2 The Zero Dynamics 


In this section we discuss an important concept that in many instances plays a role exactly similar to that 
of the “zeros” of the transfer function in a linear system. 

Given a single-input single-output system, having relative degree r < n at x = 0 represented by equa- 
tions of the form of Equation 37.3, consider the following problem, which is sometimes called the Problem 
of Zeroing the Output. Find, if any, pairs consisting of an initial state x° and of an input function u°(-), 
defined for all t in a neighborhood of t = 0, such that the corresponding output y(t) of the system is iden- 
tically zero for all t in a neighborhood of t = 0. Of course, the interest is to find all such pairs (x°, u°) and 
not simply the trivial pair x° = 0, u° = 0 (corresponding to the situation in which the system is initially 
at rest and no input is applied). 

Recalling that, in the normal form of Equation 37.3 


y(t) = §1(0), 
we observe that the constraint y(t) = 0 for all t implies 
€1(t) = Eo(t) =... =8,(1) =0, 


that is, (¢) = 0 for all t. In other words, if the output of the system is identically zero, its state necessarily 
respects the constraint &(t) = 0 for all t. In addition, the input u(t) must necessarily be the unique solution 
of the equation 

0 = b(0, n(t)) + a(0, n(t))u(t) 
[recall that a(0, n(t)) 4 0 if y(t) is close to 0]. As far as the variable y(t) is concerned, it is clear that, &(t) 
being identically zero, its behavior is governed by the differential equation 


H(t) = q(0, n(¢)). (37.7) 


From this analysis it is possible to conclude the following. In order to have the output y(t) of the system 
identically zero, necessarily the initial state must be such that €(0) = 0, whereas (0) = n° can be chosen 
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arbitrarily. According to the value of n°, the input must be set equal to the following function 


b(0, n(t)) 


wD =< One) 


where y(t) denotes the solution of the differential equation 


H(t) = q(O, n(t)) 


with initial condition n(0) = n°. Note also that for each set of initial conditions (, n) = (0, n°) the input 
thus defined is the unique input capable of keeping y(t) identically zero. 

The dynamics of Equation 37.7 correspond to the dynamics describing the “internal” behavior of the 
system when input and initial conditions have been chosen in such a way as to constrain the output to 
remain identically zero. These dynamics, which are rather important in many instances, are called the 
zero dynamics of the system. 

The previous analysis interprets the trajectories of the (n — r)-dimensional system 


h = (0, n) (37.8) 


as “open-loop” trajectories of the system, when the latter is forced (by appropriate choice of input and 
initial condition) to constrain the output to be identically zero. However, the trajectories of Equation 37.8 
can also be interpreted as autonomous trajectories of an appropriate “closed-loop system.” In fact, consider 
again a system in the normal form of Equation 37.3 and suppose the feedback control law of Equation 37.4 
is imposed, under which the input-output behavior becomes identical with that of a linear system. The 
corresponding closed-loop system thus obtained is described by Equations 37.5. If the linear subsystem 
is initially at rest and no input is applied, then y(t) = 0 for all values of t, and the corresponding internal 
dynamics of the whole (closed-loop) system are exactly those of Equation 37.8, namely, the zero dynamics 
of the open-loop system. 


Remark 37.2 
In a linear system, the dynamics of Equation 37.8 are determined by the zeros of the transfer function of 


the system itself. In fact, consider a linear system having relative degree r and let 


bo tbyst-+-+by_7~_1s" 1 + st 
ao + aist++++ayn—1s"-1 +58" 


H(s)=K 


denote its transfer function. Suppose the numerator and denominator polynomials are relatively prime 
and consider a minimal realization of H(s) 


x= Ax+Bu 
y=Cx 
with 
0 1 0 0 0 
0 0 1 0 0 
A= : B= 
0 0 0 1 0 
—ag —a, —Aa2 —Ay-| K 
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The realization in question can easily be reduced to the form of Equation 37.2. For the & coordinates 
one has to take 


&1 = Cx = boxy + bx. + +++ + Oy—r—1Xp—r + Xn 
&2 = CAx = box. + byx3 +++ ++ Un—r—1Xn—r41 + Xn—r42 


§r = CA 1x = box, cr by x41 Petes bp—1r—-1Xn—-1 + Xp. 
while for the n coordinates it is possible to choose 
NM =*1 
2 = x2 
Yun-r = Xn—-r- 


In the new coordinates we obtain equations in normal form, which, because of the linearity of the 
system, have the following structure 


& = AE+ B(RE + Sy+ Ku) 
i= P§+ Qn 


where R and S are row vectors and P and Q are matrices of suitable dimensions. The zero dynamics of 


this system, according to our previous definition, are those of 
1 = Qn. 


The particular choice of the last n — r new coordinates (i.e., of the elements of 1) entails a particularly 
simple structure for the matrices P and Q. As a matter of fact, it is easily checked that 


0 0 1 0 tee 0 

0 0 0 1 tee 0 
pS OSM. A ; io Se : 

0 Gi. “ih. SG 2 1 

1 —bo by) —bp 2+) = By r-1 


From the particular form of this matrix, it is clear that the eigenvalues of Q coincide with the zeros of 
the numerator polynomial of H(s), that is, with the zeros of the transfer function. Thus, it is concluded 
that in a linear system the zero dynamics are linear dynamics with eigenvalues coinciding with the zeros 


of the transfer function of the system. 

These arguments also show that the linear approximation, at 7 = 0, of the zero dynamics of a system 
coincides with the zero dynamics of the linear approximation of the system at x = 0. In order to see this, 
consider for f (x), g(x) and h(x) expansions of the form 


f(x) =Ax+fo(x) 
g(x) = B+ gi(x) 
h(x) = Cx + h2(x) 


eh = ees 


where 
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An easy calculation shows, by induction, that 


Lfh(x) = CAKx + d(x) 


| S 
Ox ae 


CA'B=LgL/h(0)=0 for allk<r—1 


where d;(x) is a function such that 


From this, one deduces that 


CA™'B = LgLe-'h(0) #0 


that is, the relative degree of the linear approximation of the system at x = 0 is exactly r. 
From this fact, it is concluded that taking the linear approximation of equations in normal form, based 
on expansions of the form 
b(§, n) = RE + Synt bo(§,n) 
a(n) =K + a1, n) 
q(§n) = P§ + Qn+ m(§n) 


yields a linear system in normal form. Thus, the Jacobian matrix 


) 
a=[F 
NJ (En)=0 


which describes the linear approximation at n = 0 of the zero dynamics of the original nonlinear system, 
has eigenvalues that coincide with the zeros of the transfer function of the linear approximation of the 
system at x = 0. 


37.3 Local Stabilization of Nonlinear Minimum-Phase Systems 


In analogy with the case of linear systems, which are traditionally said to be “minimum phase” when all 
their transmission zeros have negative real part, nonlinear systems (of the form of Equation 37.1) whose 
zero dynamics (Equation 37.8) have a locally (globally) asymptotically stable equilibrium at z = 0 are 
also called locally (globally) minimum-phase systems. As in the case of linear systems, minimum-phase 
nonlinear systems can be asymptotically stabilized via state feedback. We discuss first the case of local 
stabilization. 

Consider again a system in normal form of Equation 37.3 and impose a feedback of the form 


1 
u= (—b(§, n) — co&1 — €1§2 —- -- — cr-1& r) (37.9) 
a(§,n) 
where Co, Cj, ++, C;— 1 are real numbers. 
This choice of feedback yields a closed-loop system of the form 
E = (A+ BK 
: ( % (37.10) 
n= (6) 
with 
0 1 0 0 
0 0 1 0 
A+BK= 
0 0 0 1 
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In particular, the matrix A + BK has a characteristic polynomial 
pls) =coteyst---+ cps” +8". 


From this form of the equations describing the closed-loop system we deduce the following interesting 
property. 


Proposition 37.1: 


Suppose the equilibrium yn = 0 of the zero dynamics of the system is locally asymptotically stable and all 
the roots of the polynomial p(s) have negative real part. Then the feedback law of Equation 37.9 locally 
asymptotically stabilizes the equilibrium (€, n) = (0, 0). 


This is a consequence of the fact that the closed-loop system has a triangular form. According to a 
well-known property of systems in triangular form, since by assumption the subsystem 


7 = q(0, n) 


has a locally asymptotically stable equilibrium at y = 0 and the subsystem 


§ = (A+ BK)é 


has a (globally) asymptotically stable equilibrium at & = 0, the equilibrium (&, n) = (0,0) of the entire 
system is locally asymptotically stable. 
Note that the matrix 


Oq > 
C (E,n)=(0,0) 


characterizes the linear approximation of the zero dynamics at n = 0. If this matrix had all its eigenvalues in 
the left complex half-plane, then the result stated in Proposition 37.1 would have been a trivial consequence 
of the Principle of Stability in the First Approximation, because the linear approximation of Equation 37.10 


has the form 
. 5 


However, Proposition 37.1 establishes a stronger result, because it relies only upon the assumption 
that n = 0 is simply an asymptotically stable equilibrium of the zero dynamics of the system, and this (as 
is well known) does not necessarily require, for a nonlinear dynamics, asymptotic stability of the linear 
approximation (i.e., all eigenvalues of Q having negative real part). In other words, the result in question 
may also hold in the presence of some eigenvalue of Q with zero real part. 

In order to design the stabilizing control law there is no need to know explicitly the expression of the 
system in normal form, but only to know the fact that the system has a zero dynamics with a locally 
asymptotically stable equilibrium at 1 = 0. Recalling how the & coordinates and the functions a(€, n) and 
b(&, n) are related to the original description of the system, it is easily seen that, in the original coordinates, 
the stabilizing control law assumes the form 

1 


u= Lg Ly *h(x) ( Lyh(x) — coh(x) — er L¢h(x) — ++ — cL} "h(x)) 


which is particularly interesting because expressed in terms of quantities that can be immediately calcu- 
lated from the original data. 
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If an output function is not defined, the zero dynamics are not defined as well. However, it may 
happen that one is able to design a suitable dummy output whose associated zero dynamics have an 
asymptotically stable equilibrium. In this case, a control law of the form discussed before will guarantee 
asymptotic stability. This procedure is illustrated in the following simple example, taken from [5]. 


Example 37.1: 


Consider the system 
xy = x?x3 
XxX =Xo+u 


whose linear approximation at x = 0 has an uncontrollable mode corresponding to the eigenvalue 
}. = 0. Suppose one is able to find a function y(x;) such that 


4 =x? Lyoa)P 


is asymptotically stable at x; = 0. Then, setting 
y = h(x) = yx) — x2 

a system with an asymptotically stable zero dynamics is obtained. As a matter of fact, we know that 
the zero dynamics are those induced by the constraint y(t) = 0 for all t. This, in the present case, 
requires that the x; and x2 respect the constraint 

yxq) —x2 =0. 
Thus, the zero dynamics evolve exactly according to 

X= x7 Lyon) P 
and the system can be locally stabilized by means of the procedure discussed above. A suitable 
choice of y(x7) will be, e.g., 

y(x1) = —x, . 


Accordingly, a locally stabilizing feedback is the one given by 


a Dp _ see ee _ 2.3 
a) = Toa! Leh(x) — ch(x)) = —og — (1+ 0x2 — x7x5 


with c > 0. 


37.4 Global Stabilization of Nonlinear Minimum-Phase Systems 


In this section we consider a special class of nonlinear system that can be globally asymptotically stabilized 
via state feedback. The systems in question are systems that can be transformed, by means of a globally 
defined change of coordinates and/or feedback, into a system having this special normal form 


z= fo(z,§1) 
f= 
hep (37.11) 
Ep = e, 


Ey =U. 
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Remark 37.3 


Note that a system in the normal form of Equation 37.2, considered in the previous sections, can indeed 
be changed, via feedback, into a system of the form 


£1 =§ 

Ep) =&, (37.12) 
E, =u 
i= 4(&, 0). 


Moreover, if the normal form of Equation 37.2 is globally defined, so also is the feedback yielding 
the (globally defined) normal form of Equation 37.12. The form of Equation 37.11 is a special case of 
Equation 37.12, the one in which the function q(&, 1) depends only on the component & of the vector 
&. In Equation 37.11, for consistency with the notations more frequently used in the literature on global 
stabilization, the vector z replaces the vector y of Equation 37.12 and the places of z and & are interchanged. 

In order to describe how systems of the form of Equation 37.11 can be globally stabilized, we begin 
with the analysis of the (very simple) case in which r = 1. For convenience of the reader, we recall that a 
smooth function V : R” — R is said to be positive definite if V(0) = 0 and V(x) > 0 for x £ 0, and proper 
if, for any a € R, the set V—!((0, a]) = {x € R":0 < V(x) <a}is compact. 

Consider a system described by equations of the form 


2 =f(@5) (37.13) 
=u 

in which (z, &) € R” x R, and f(0, 0) = 0. Suppose the subsystem 
z= f(z,0) 


has a globally asymptotically stable equilibrium at z = 0. Then, in view of a converse Lyapunov theorem, 
there exists a smooth positive definite and proper function V(z) such that ay f(z, 0) is negative for each 
nonzero z. Using this property, it is easy to show that the system of Equation 37.13 can be globally 
asymptotically stabilized. In fact, observe that the function f(z, €) can be put in the form 


f(z,§) =f (Zz 0) + plz, §)§ (37.14) 


where p(z, €) is a smooth function. For it suffices to observe that the difference 


f(z, 8) =f(z,§) —f(z, 0) 


is a smooth function vanishing at £ = 0, and to express f(z, ) as 


zr, a, [) OF s8), | Od) 


Now consider the positive definite and proper function 


W(z,&) = Viz) + xe , (37.15) 


and observe that 


dW AW) (f(z,8)\ _ aV av 
(BE) ES) Ben Sere 
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Choosing 


OV 
u=u(z,&)=—€E Dz p(z,&) (37.16) 


aw aw) (fle®)) 4 
dz & } \u(z,&) 
for all nonzero (z, €). By the direct Lyapunov theorem, it is concluded that the system 


Z=f(z,&) 


§ = u(z,&) 
has a globally asymptotically stable equilibrium at (z, &) = (0, 0). 

In other words, it has been shown that, if z = f(z, 0) has a globally asymptotically stable equilibrium 
at z = 0, then the equilibrium (z,&) = (0,0) of the system of Equation 37.13 can be rendered globally 
asymptotically stable by means of a smooth feedback law u = u(z, €). 

The result thus proven can be easily extended by showing that, for the purpose of stabilizing the 
equilibrium (z, &) = (0, 0) of Equation 37.13, it suffices to assume that the equilibrium z = 0 of 


z=f(z,&) 


is stabilizable by means of a smooth law — = v*(z). 


yields 


Lemma 37.1: 


Consider a system described by equations of the form of Equation 37.13. Suppose there exists a smooth 
real-valued function 

§=v"(z), 
with v* (0) = 0, and a smooth real-valued function V(z), which is positive definite and proper, such that 


OV . 
pr (z)) <0 


for all nonzero z. Then, there exists a smooth static feedback law u = u(z, &) with u(0, 0) = 0, and a smooth 
real-valued function W(z,&), which is positive definite and proper, such that 


dW AW\ (f(z,8) 
(G ) ea) = 


In fact, it suffices to consider the (globally defined) change of variables 


for all nonzero (z, &). 


y='—v*(2), 
which transforms Equation 37.13 into 
z=f(z,v*(z)+y) 
av* (37.17) 
y= a feve+y)t+u, 
and observe that the feedback law 
0 Fe: 
u= Ste v*(z)+y)+ul 


changes the latter into a system satisfying the hypotheses that are at the basis of the previous construction. 
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Using repeatedly the property indicated in Lemma 37.1, it is straightforward to derive the following 
stabilization result about a system in the form of Equation 37.11. 


Theorem 37.1: 


Consider a system of the form of Equation 37.11. Suppose there exists a smooth real-valued function 
§i=Vv*(z), 
with v* (0) = 0, and a smooth real-valued function V(z), which is positive definite and proper, such that 
ov x 
sz —Iolz, v"(z)) <0 
jz Jol v"2)) 
for all nonzero z. Then, there exists a smooth static feedback law 


u=u(z,&),...,&,) 


with u(0,0,...,0)=0, which globally asymptotically stabilizes the equilibrium (z,&1,...,€)= 
(0,0,...,0) of the corresponding closed-loop system. 


Of course, a special case in which the result of Theorem 37.1 holds is when v*(z) = 0, that is, when 
Z = fo(z, 0) has a globally asymptotically stable equilibrium at z = 0. This is the case of a system whose 
zero dynamics have a globally asymptotically stable equilibrium at z = 0, that is, the case of a globally 
minimum-phase system. 

The stabilization procedure outlined above is illustrated in the following example, taken from [1]. 


Example 37.2: 


Consider the problem of globally asymptotically stabilizing the equilibrium (x1, x2,x3) = (0, 0,0) of 
the nonlinear system 


XI =x} 
X =x? (37.18) 


To this end, observe that a “dummy output” of the form 
y = X3 — V*(x1,x2) 


yields a system having relative degree r = 1 at each x € R? and two-dimensional zero dynamics. The 
latter, that is, the dynamics obtained by imposing on Equation 37.18 the constraint y = 0, are those 
of the autonomous system 


(37.19) 
X= (V* (x7, x9). 


From the discussion above we know that, if it is possible to find a function v*(x1, x2) that globally 
asymptotically stabilizes the equilibrium (x), x2) = (0,0) of Equation 37.19, then there exists an input 
u(x), X2, x3) that globally asymptotically stabilizes the equilibrium (x1, x2, x3) = (0, 0, 0) of Equation 37.18. 
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It is easy to check that the function 
v* (x1, x2) = —x1exp(x1X2) 


accomplishes this task. In fact, consider the system 
. 3 
xX, =X 
ne (37.20) 
X2 = —xjexp(3x1Xx2), 


and choose a candidate Lyapunov function 
V(x1,2%2) = a +x5 : 
which yields 
V= A(xyx2)°(1 — exp(3x1x2)). 


This function is nonpositive for all (x1, x2) and zero only at x; = 0 or x2 = 0. Since no nontrivial trajectory 
of Equation 37.20 is contained in the set 


M = {(x1,%2):V = 0}, 
by Lasalle’s invariance principle it is concluded that the equilibrium (x, x2) = (0, 0) of Equation 37.20 is 
globally asymptotically stable. 

In order to obtain the input function that globally stabilizes the equilibrium (x1, x2, x3) = (0,0, 0) of 
Equation 37.18, it is necessary to use the construction indicated in the proof of Lemma 37.1. In fact, 
consider the change of variables 

y = x3 — V*(x1, 2) 
which transforms Equation 37.18 into 


xy = 


4 = (y+ *(x1,%2))° (37.21) 
av* we ov* 
Ox 2 Ox2 


(y +v*(x1,%2)). 


Choosing a preliminary feedback 


i= ee ‘ie (y+v*(x1,%2))? + 
yields 
x= ie 
i = (y +" (x1,.x2))? (37.22) 
ya’. 
which has exactly the form of Equation 37.13, namely, 
z= f(z,8) 
g=u', 
with 


3 
x) 


0 
F128) (saeaten®) +8 (apeenmn?+aotta, ane +4) 


and z = f(z, 0) has a globally asymptotically stable equilibrium at z = 0. As a consequence, this system can 
be globally asymptotically stabilized by means of a feedback law u’ = u'(z, &) of the form of Equation 37.16. 
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38.1 Introduction 


Time-dependent events a(t) and b(t) are said to commute if the occurrence of a(t) during a time interval 
T followed by the occurrence of b(t) during an interval S leads to an outcome that does not change when 
the order of events is reversed. Denoting by b(S)a(T) the occurrence of a followed by b, then a(T)b(S) 
denotes the reversed order, and a and b commute if a(T)b(S) = b(S)a(T). Otherwise, a and b are said to 
be noncommutative events. 

Most events do not commute, as is evident from common experience. For instance, filling the swim- 
ming pool with water and then diving into the pool results in a state different from one in which the order 
is reversed. Driving an automobile relies on a more subtle use of noncommutativity (based on nonholon- 
omy). Any automobile driver knows that the rotations of the steering wheel do not commute with either 
forward or backward motions of the automobile. The ability to park the automobile in a tight spot, a most 
demanding challenge for an inexperienced driver, is only possible because of the noncommuting nature 
of these events. An experienced driver knows that it is possible to execute a parallel displacement of an 
automobile in any space which is large enough to allow some forward and backward movement, although 
the number of maneuvers may be so large that it might very well be advisable to look for another parking 
spot with more space to spare. 

Control of many dynamic systems, like the control of an automobile, consists of a time-sequential 
application of noncommuting events. Knowing which event to apply at a given time is a basic require- 
ment of successful control. The recognition of noncommutativity as a fundamental issue of control is a 
starting point for geometric control; this chapter describes the main mathematical tools required for its 
understanding. 
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38.2 Notations and Basic Assumptions 


The subsequent discussion is confined to control systems described by a differential equation a = F(x, u) 
in which the control functions u(t) = (u(t), ..., Um(t)) take values in a fixed set U in R™. Most of the 
theory presented in this chapter is extracted through F and its derivatives, and for that reason it is 
expedient to assume that the state variable x(t) belongs to an analytic manifold M and that F(x, u) is 
an analytic vector field for each u in U. The reader not familiar with these notions may assume at the 
beginning that M is a finite dimensional vector space and that for each u € U, F(x, u) can be represented 
by its Taylor series at each point x in M. The extensions to arbitrary manifolds will be defined as 
needed. 

Throughout this chapter A(x, T) will denote the set of states reachable from an initial state x in 
exactly T units of time. Then, A(x,<T)= (J A(x,t), and A(x) = LU A(x, t). As will be explained 

O<t<T t>0 

later, a differential control system may also ‘be viewed as a family of vector fields F, in which case 
the reachable sets will be subscripted A¢(x, T), Ar(x, < T), and A(x) in order to emphasize their 
dependence on a given system ¥. These notations will be needed particularly when discussing several 
systems simultaneously. 


38.3 Vector Fields 


Geometric control theory begins with a distinctive view of a differential equation, initiated by H. Poincaré 
in the latter part of the 19th century, that the solutions of 


“ =F(x(t)) x(t)inM (38.1) 


can be analyzed in simple mathematical terms without ever having to solve the differential equation. The 
corresponding theory is based ona single assumption that for each initial state x differential Equation 38.1 
admits a unique solution through that state defined for all times t. Assuming that this is the case, let x(xo, f) 
denote the solution of Equation 38.1 for which x(xo, 0) = xo. The mapping (xo, t) > x(xo, f) is called the 
flow, or dynamic system induced by the vector field F. Its essential properties are 


1. x(x, 0) = xo for each xo in M 
2. x(xo,t +s) = x(x(x0, t), s)) = x(x(x0, 5), f) for all xo in M ands,tinR 
3. 2x(xo, t) = F(x(xo, t)) 


At each instant of time, t, the flow of F induces a transformation on M which will be denoted by exptF: 
exptF maps each initial point x9 onto x(x9,t). It follows from (1) and (2) that exp OF = Identity and 
that exp(t + s)F = (exptF)(expsF) = (expsF)(exptF) for any s and t, with (expsF)(exptF) denoting the 
composition of mappings. Since exp 0F = Identity, it follows that (exptF)~! = exp —tF, and therefore, 
the mappings {exptF : t € R} form a commutative group. This group is called the one-parameter group 
of transformations induced by F. 


Example 38.1: 


If F is a linear vector field, i.e., F(x) = Ax for some linear mapping on M then x(xo, t) = elAxy with 


k 
eA — paar gat. Thus, in this case exptF is a linear transformation on M equal to the exponential 
of a linear mapping. 
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Example 38.2: 


If F(x) = b is a constant vector field, then x(xo, t) = xq + tb, and therefore each transformation exptF 
is a translation in the direction of b. 


Vector fields, no matter how nonlinear, admit linear interpretations on an infinite dimensional vector 
space in which exptF becomes the exponential of a linear mapping. In this interpretation vector fields act 
linearly on functions on M as follows: let 


(exptF)(f) (xo) =f (x(xo, t)) 
for any real-valued function, f. Then Ff is the function on M defined by 


d 


Ef = = (exptF)(f)li=0 


Ff is called the derivation of f along the vector field F. 
Ff admits a simple description in any system of coordinates on M. Assuming that M is a linear vector 


space, then any basis aj,..., dy, in M induces coordinates x), ...,x, on M by the formula x = sj Xjdj. 
Then F(x) = eran F;(x1,...,Xy)aj and Equation 38.1 is written as a system of differential equations in R” 
dx; 
Fem Fil 9 %n) Fels. sshis (38.2) 


Any real-valued function on M becomes a function on R” by the correspondence f(x) =f (x1,... Xn). 
Then, 


n 


d 
F fers. sala = > Ge. sad Fies 5%) 


iat O%i 
and therefore 


“9 
(BAG os sk =) BGs 50) 


Ox 
jy 


It follows that (Ff )(x) is equal to the directional derivative of f at x in the direction of F(x). In particular, 
when f = x;, then Fx; = Fj,i=1,...,n. Evidently F acts linearly on functions (as a directional derivative) 
and therefore satisfies further properties: 

Ff = 0 for constant functions f, and F(fg) = f (Fg) + g(Ff) with respect to the products of functions. 
(fg denotes the product of f and g, ie., (fg)(x) =f (x)g(x) and f (Fg) is the product of f with Fg. Note that 
f(Fg) 4 (Ef)g.) 


As a mapping on the space of functions exptF satisfies 


d d" 
nope = F(exptF) = (exptF)F and qn OPP = F"exptF (38.3) 
with each product denoting the composition of linear mappings. Analytic vector fields can be rep- 


resented by their Taylor series and therefore (exptF)(f) = )-79 PK ) for any analytic function 


f. The reader can easily verify that in each system of coordinates, F?f = F(Ff) = F (Oka afi) = 


2 
ijel (sake + # je). Then Ff is the directional derivative of F7f in the direction F and so on 
iOXj i jf 


for each derivate F"f. 
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38.4 The Lie Bracket 


With these notions and notations at our disposal, let us return to the commutativity issue raised in the 
introduction, and consider commutativity properties of exptF and expsG corresponding to vector fields 
F and G. This question may be further motivated through the following control theoretic context: 


Consider a control system 
dx 
Fe MOP) +. — ue) GEC) 
with switching control u which can only take values 0 or 1 (on-off controls). During any time interval that 
the control is turned on the system follows F and during the time intervals that the control is switched off 
the system follows G. 


Let u(t) and u(t) denote the following control functions defined in an interval [0, T + S]: 


1, te€[0,T) 
no=| 
0, te[T,T+S] 


Be 0, te€[0,S) 
eV tel. T+S] 


The corresponding trajectories x(t) and x(t), both initiating from the same initial point xo, are 
given by: 


x(t) = (exptF)(xo), t € [0, T) 

x(t) = (exp(t — T)G)(exptTF)(xo), t¢€[T,T+S] 
and 

x2(t) = (exptG)(xo), t € [0, S) 

x2(t) = (exp(t — S)F)(exp SG)(xo) for t € [S,S+ T] 


At the terminal time t = T + S, x; (t) = (exp SG) (exp TF)(xo), and x(t) = (exp TF) (exp SG)(xo). 
Assuming that the control actions of u(t) and u(t) lead to the same terminal state independently 
of the initial point xo and of the switching times S and T, then (exp SG)(exp TF) = (exp TF) (exp SG). 
This equality remains unaltered when the domain is extended to the space of functions and therefore, 
2 2 (expsG)(exptF)(f) = 2 2 (exptF)(expsG)(f) for any function f. 
Taking advantage of formulas (Equation 38.3) 
00 


1 5, CxPSG) exptP if = © GlexpsG)(expt)(f) = G(expsG)(FexptF)f, 


and 
° o (expt ylexpsG)(f) = F(exptF)(GexpsG)f 


Evaluating these derivatives at t = s = 0 gives 


G(Ff) = F(Gf) 


Definition 38.1: 


Let F and G be any vector fields on M. Then [F, G](f) = G(Ff) — F(Gf) for any real-valued function f on 
M. [F, G] is called the Lie bracket of F with G. 


The Lie Bracket and Control 38-5 


It follows that [F, G] is a vector field for any vector fields F and G. Note that [F, G] = —[G, F]. The ith 
coordinate of the Lie bracket [F, G] is given by [F, G](x;). Therefore, 


n 


[F, Gl(x;) = GF) — FG) = >> 


Jai 


OF; OG; 


—G;, —- —F; 38.4 
Ox; s Ox; ( ) 


The calculations above show that if (expsG)(exptF) = (exptF)(expsG) for all s and t then [F, G] = 0. 
Somewhat remarkably, the converse is also true; ie., if [F, G] = 0, then (exptF)(expsG) = (expsG)(exptF). 


Example 38.3: 


If F(x) =6 and G(x)=c are any constant vector fields, then their coordinates are constant 
functions, and therefore [F,G]=0 (as can be seen from Equation 38.4). Then, (exptF)(x) = 
x+tb and (expsG)(x)=x-+sc, and therefore (exptF)(expsG)(x) = (x +sc)+ tb=(x+tb)+sc= 
(expsG)(exptF)(x), confirming that the flows commute. 


Example 38.4: 

Let F(x) = Ax be a linear vector field and G(x) = b a constant vector field. In terms of any linear 

coordinates x1,...,Xn,Fj(x1,...,Xn) = yet Ajj - xj- (Ajj) is the matrix of A relative to this basis. Then 
"OF; 


dG; 7 7 
[F, G](xj) = mo og ee 
j=l j=! 


j=| 


Thus [G, F] is a constant vector field equal to Ab. [G, F] = 0 if and only if b € ker A. 


0 -1 O 0 
For instance, the rotation A = ( 0 | and b= (°) commute, while A does not commute 
0 O 0 1 


1 
with b = | 0 | (Figure 38.1). 
0 


(b) 


*) 


FIGURE 38.1 Commutativity: (a) Commuting case and (b) Noncommuting case. 
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Example 38.5: 


Let F(x) = Ax and G(x) = Bx be linear vector fields on M. Then [F, G] is a linear vector field given by 
[F, G](x) = (AB — BA)(x). 


For example, ifA = (; . 


0 a) andB= ¢ 4 are the matrices corresponding to F and G (relative 


0 


to a linear system of coordinates) then C = 2 @ 


3) is the matrix that corresponds to [F, G]. The 


flows of these fields are shown in Figure 38.2. 


(G) 


(F, G) 


FIGURE 38.2 Lie brackets of vector fields. 
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Return now to the general discussion, and consider the expression 
(exp —tG)(exp —tF)(exptG)(exptF) (38.5) 


As shown earlier, this expression reduces to the identity when [F, G] = 0. Begin ig a simple case with 
F and G linear vector fields defined by linear mappings A and B. Then exptF = e4 = )°?o. 0k Z GA‘, and 


exptG = rye 9 e Be, We shall be interested only in the terms up to and including the second-order terms 
in t. Then, 


t2 
eel 14 t(A+B)+¢BA+ ris +B?) 


and 


t2 
e 8 e-4# = 1 = A+ B+ PBA+ rie +B?) 


Therefore, 


eo Be tA GIB tA _ eae. “(a +B) (141445) 478A +5 ee 


=14H(A+B)4+PBA+4 — (A +) — H(A+B) — P(A +8)? +PBA4 F(a? + BF) 


=1—t?(AB+BA)+ 5 =1+t?(BA— AB) 
These calculations show that up to second-order terms in f, 
(exp —tG) (exp —tF)(exptG)(exptF) = I + t7[G, F] 


This result extends to arbitrary vector fields, as can be demonstrated through the formulas exptF = 
pyar, UK ) and (exptG)(f) = > -e29 EGE ) described earlier. These arguments show that the proof 
for the general flows is the same as it is for the linear flows. 

The asymptotic formula obtained above shows that the curve o(t) = (exp —4/tG)(exp —./tF) 
(expt /tG)(exptV/tF)(xo) satisfies o(0) = xo and 22 a (0) = [G, F](xo). 

The preceding developments may be stated in a Donel theoretic context, as follows. 


Theorem 38.1: 


Suppose that ae = F(x, u) is any system in M with the control functions taking place in a set U C R™. 
Suppose that F\(x) = F(x, u,) and Fy(x) = F(x, uz) for some choices u, and uz of control values. Assume 
that there exist control values u3 and ug in U such that —F,(x) = F(x, u3) and —G(x) = F(x, ug) for all x. 
Then there is a curve o(t) contained in the reachable set A(x, < €) for anye > 0 such that 7 de 7 (0) = [Fi, Fo] (x). 
That is, the system can move infinitesimally in the direction of the Lie bracket. 


Example 38.6: 
Oe 1 0 
LetA= 1 0 and B= na define the bilinear system = (1 — u)Ax + uBx in M = R*, with 


0 < u(t) < 1. Then, u= 0 follows F(x) = Ax and u = 1 follows ae = Bx. The Lie bracket [F,G] is a 
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LE, G](«) 


*1 


A(x, <€) 


FIGURE 38.3 Noncontrollability of the Lie bracket. 
P ' : s : 0 I\.. 
linear field given by the rotation matrix 2 1 of Since 


dx, 
ran ux; +(1—u)x2 and aE 


dx. 
= —x2u+x1(1—u), 


oo (t)x2(t) = (uxy + (1 — u)xa)x2 + (—xQu + x4 (1 — u))x1 = (1 — u(x? + x3) > 0. 


Thus, x; (t)x2(t) is increasing for any admissible control function. The system cannot move infinitesimally 
in the direction of [F, G] because neither — F nor —G can be traced by the admissible controls (Figure 38.3). 


38.5 The Lie Algebras 


From a geometric point of view, a control system is a family of vector fields parameterized by controls. 
Each control value determines a vector field, and the corresponding trajectory is a solution curve of this 
field. As the control switches to a new value so does the vector field and the trajectory begins to follow 
the direction of the new field. From this view a trajectory generated by a piecewise constant control is a 
continuous curve in M having discontinuous derivatives with the breaks in the derivative corresponding 
to the changes in vector fields caused by the switches in the control. Such curves are conveniently called 
continuous broken trajectories. Between any consecutive breaks, the trajectory is a solution curve of a 
vector field in the family. 

From this perspective, a linear control system a = Ax + Bu, u € U isa collection of affine vector fields 
F of the form F,,(x) = Ax + Bu with u € U. An affine vector field F is any vector field in M which satisfies 
F()> 4ixi) = Y> X;F(x;) for any affine combination )> d;x; with > 4; = 1. It can be shown that any affine 
field is the sum of a linear vector field and a constant vector field. The exponential map exptF of any 
affine vector field F(x) = Ax + b is given by (exptF)(x) = et (x + is e “5b ds). For instance, if b € ker A, 
then e~45b = b, and (exptF)(x) = e4!x + bt. 
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Vector fields form a vector space under pointwise addition and multiplication by scalars. The solution 
curves of F and XF, da real number, differ only by reparameterization of time; ie., if ae (t) = F(x(t)) then 
# x(n) = \F(x(At)), and consequently y(t) = x(t) is a solution curve of XF. 


For sums of vector fields the situation is different. Thus, F(x) = i; i) @ , G(x) = (; 5) ie 


0 O x2 1 0 x2 
add to Hi) = i; i) a) 
2; 


The phase portraits of these flows are shown in Figure 38.4. 

It is known that every solution curve of the sum F + G can be approximated by a curve that oscillates 
between the solution curves of F and the solution curves of G, as shown in Figure 38.5. 

The vector space structure, together with the Lie bracket operation, turns the set of all vector fields into 
an algebra called the Lie algebra. 

Any set of vector fields F generates the smallest sub-algebra that contains F. We will use Lie(F) to 
denote the Lie algebra generated by a family of vector fields F. 


) (G) 


(F+G) 


FIGURE 38.4 Sums of vector fields. 


38-10 Control System Advanced Methods 


FIGURE 38.5 Chattering along the sum of vector fields. 


Example 38.7: 


Let F consist of two vector fields F and G in R2 given by Ff = x? OF and Gf = xe x. Then, [F, G]f = 


—2x? fe. Furthermore, [F, [F, G]] = 6x? a. It then follows by induction that each vector field xt oe 


is contained in Lie(F), and therefore Lie(F) is an infinite dimensional algebra of vector fields. 


Example 38.8: 


Consider now the family of affine vector fields F = {Fy(x) = Ax + Bu: u € U} defined by a linear 
system with a constraint set U. When the linear span of points in U spans R™, Lie(F) coincides with 
the Lie algebra generated by Fg(x) = Ax and the constant vector fields with values in the range of B. 


As we have already shown in Example 38.2, [Fo,G] = Ab for any constant vector field G = b. Let B 
denote the range space of B. Then, it follows by the preceding observation that the space of constant 
vector fields in B + AB is contained in Lie(¥). Denote by £; the vector space of all constant vector fields 
with values in B+ AB+---+A*—!B. It follows that Lyi = [Fo, £4] +L, and therefore each space 
Lx, is contained in Lie(F). Since each power A”tk is a linear combination of {A*:0<k<n—l}, asa 
consequence of the Cayley-Hamilton theorem, it follows that Lie(¥) is equal to the linear span of Fy and 
Ly, and is therefore a finite dimensional Lie algebra. The space Ly is usually described as the range space 
of the controllability matrix 

(BAB...A"—!B) 


An arbitrary family of vector fields F in Lie(¥), when evaluated at each point x, defines a linear 
space of tangent vectors at x. For instance, in Example 38.7, each vector in Lie(¥) evaluated at x; = 0 
is equal to zero. Therefore, each element in Lie(¥) is equal to zero at such points, and consequently the 
corresponding space of tangent vectors is zero-dimensional. At any other point of the plane G and F are 
linearly independent, and their span is two-dimensional. 
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The Lie algebra in Example 38.8 evaluated at the origin is equal to the range space of the controllability 
matrix. The Lie algebra evaluated at any other point is the vector sum of Ax and the range of the 
controllability matrix. 

We shall use Lie,(F) to denote the vector space of all tangent vectors F(x) with F in Lie(F). Then 
dim Lie, (F) will denote the dimension of this vector space. 

The following theorem is of fundamental importance for geometric control theory. 


Theorem 38.2: 


Let F be any family of vector fields, and let Ax(x, < T) denote its reachable set from x in T units of time (in 
accordance with the conventions outlined earlier). Then dim Liex(F) = dim M is a necessary and sufficient 
condition that A¢(x, < T) has anon-empty interior in M. Furthermore, when dim Lie,(M) = dim M, then 


clA#(x,< T) C intA¢(x,< T+¢) CAzr(x,< T+) foranyT>0 and anye>0. 


In the preceding notation, cl(A) denotes the topological closure of a set A, which means that cl(A) 
consists of A along with all points of M which are limit points of elements in A. The interior of any set 
A, denoted by int (A), consists of all points a in A contained in an open ball in M centered at a which is 
entirely contained in A. 

Typically, the initial point x belongs to the boundary of the reachable set A(x, < T), as in Example 38.6. 
x is said to be small-time controllable by F whenever x belongs to the interior of A¢(x, < T) for any 
T > 0. The following example shows that x can be small-time locally controllable even for families of two 
vector fields neither of which vanishes at x. 


Example 38.9: 
Let F be the family consisting of linear vector fields F and G in R2 described by A = G _) and 
B= i Ss) The solution curves of F are hyperbolas ae —x? = const while the integral curves 


of G are parabolas x7 = Or The curves are tangent to each other along the lines x; = +./2x>. Any 
initial point x along such a line is in the interior of Az (x, < T), as Figure 38.6 shows. 


For linear systems, any trajectory x(t) which originates at x» =0 is of the form x(t)= 
eAt 4 e~4sBu(s) ds, and is therefore necessarily contained in the range space of the controllability matrix, 
as can be seen from the expression 


(t—s)k 

A(t—s) k 

e€ Bu(s) = 2 7 A*“Bu(s) ds 
=0 


confirming the necessity of the Lie algebraic criterion stated in Theorem 38.2. 


38.6 The Lie Saturate 


We now shift to the invariance properties of control systems using the closure of the reachable sets as the 
basic criterion for invariance. Let F, and F2 be any families of vector fields. Then F; is said to be strongly 
equivalent to F> if 
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ee 


eA 


FIGURE 38.6 Small-time local controllability: (a) solution curves of G; (b) solution curves of F; and (c) closed 
cycles. 


(a) 


Dx 


1. Liex(F ) = Lie,(F2) for all x in M 
2. clAF, (x, < T) =clAg,(x, < T) for all T > 0 and all x in M 


Fy and Fy are said to be equivalent if (2) is replaced by clAF, (x) = clAx, (x). 


Definition 38.2: 


The (strong) Lie saturate of a given control system is the largest family of vector fields (strongly) equivalent 
to F. The (strong) Lie saturate will be denoted by LS;(F), and LS(F) will denote the Lie saturate of F. 


Definition 38.3: 


A control system F is said to be strongly controllable if A¢(x, < T) = M for each x in M and each T > 0. 
It is said to be controllable if Az (x) = M for each x in M. 
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Theorem 38.3: 


i. A control system F is strongly controllable if and only if the strong Lie saturate of F is equal to Lie(F). 
ii. F is controllable if and only if the Lie saturate of F is equal to Lie(F). 


The above theorem is called the Lie saturate criterion of controllability. As elegant as the criterion 
seems, its practical value rests on the constructive means of calculating the Lie saturate. The next theorems 
describe permissible system enlargements, prolongations, which respect the reachable sets and may be 
used to generate a procedure for calculating the Lie saturate. We say that F> is a prolongation of Fj, if 
Fy, C Fo and if F; and F, have the same Lie saturates. 

We begin by noting that the convex hull of any family of vector fields is contained in the strong Lie 
saturate of the family. Recall that the convex hull of F consists of all convex combinations 7", \iFi, 
with \7" |) 4; = 1,4 = 0, and each F; in F. It is known that 


(exp (1 > riFi)) (x) € clA (x, t) 


for each t > 0, because any trajectory of a convex combination can be approximated by a trajectory 
of ¥. The approximation is achieved by switching sufficiently fast along the trajectories of F1,..., Fm 
around the trajectory of the convex sum (as illustrated in Figure 38.5). It may happen that the termi- 
nal point of the trajectory of the convex sum is not reachable by the original system, as the example 
below shows. 


Example 38.10: 


The point x = 1, y = 0 cannot be reached in time t = 1 from the origin by the trajectories of 


—=-y*+1 


d 
= =u withu(t)=+1 


The convex hull of U = {—1, 1} is the closed interval —1 < u < 1. Therefore, u(t) = 0 is in the convex 
hull of U = {—1, 1}. The corresponding trajectory which originates at 0 is given by x(t) = t, y(t) =0 
and reaches x = 1,y=Oatt=1. 


Having taken the closure of the reachable sets as the criterion for equivalence, it becomes natural to 
pass to topologically closed families of vector fields. The choice of topology for the space of vector fields 
is not particularly important. In this context it is convenient to topologize the space of all vector fields 
by the C® topology on compact subsets of M. Rather than going into the mathematical details of this 
topology, let us illustrate the use with an example. 

Suppose that Xj (x) = (Ax + +b) is a family of affine vector fields parameterized by X. For each } 
0, (exptX,,)x = e 4x4 i e"—9Ab ds, limy_50(exptX3)(x) = x + bt because lim,_,9 e’*4 = I uniformly 
in t. Thus the limiting curve x + bt is equal to (exptXo)(x) with Xo = lim,_.9 X). 

It can be shown in general that if a sequence of vector fields converges to a vector field F then each 
curve o,,(t) = (exptF,)(xo) converges uniformly in f to o(t) = (exptF)(xo). Therefore, each family F may 
be prolonged to its topological closure. 

In addition to the convexification and the topological closure, there is another means of prolonging a 
given family of vector fields based on reparameterizations of trajectories. 
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Note that y(t) = x(Xt) remains in the reachable set A(x, < T) for any trajectory x(t) of F for which 


x(0) = xo provided that 0 < i < 1. y(t) € A(x) for any > 0. Thus, XF € £S;(F) forany0 < > < land 
any F in £S,(F). It will be useful for further references to assemble these prolongations into a theorem. 


Theorem 38.4: 


i. The Lie saturate of any system is a closed convex cone, i.e., ) i, \iFi € LS(F) for any vector fields 
F,,...,Fm in LS(F) and any numbers \, > 0,...;m > 0. 

ii. The strong Lie saturate of any family of vector fields is a closed convex body, i.e. \-i2, hiF; € 
LS;(F) for any elements Fi,..., Fm in LS;(F) and any non-negative numbers d1,...,n such that 


Ver Sh 


We now describe another operation which may be used to prolong the system without altering its 
reachable sets. This operation is called the normalization of the system. 

An invertible map ®:M— M is called a strong normalizer for F if ®(Az(®7!(x), < T)) C 
clA#(x, < T) for all x in M and T > 0. ® is called a normalizer for F if bA¢(®~!(x)) C dAg¢(x). It may 
be also said that ® is a strong normalizer if both ®(x) and ®—!(x) are contained in cLA¢(x, < T) and that 
® is a normalizer if both (x) and ®~!(x) belong to clA¢(x). In this notation ©(Az(07!(x), < T)) is 
equal to the set of points ®(y) with y belonging to. Ax (®~!(x), < T). If ® is any invertible transformation, 
and if F is any vector field then (©)(exptF)®~! is a one-parameter group of transformations and is itself 
generated by a vector field. That is, there is a vector field G such that 


(exptG) = ®(exptF)®~! 


It can be shown that G = (d®)F(®~!) where d® denotes the derivative of ©. We shall use ®y(F) to 
denote the vector field (d®)(F®~!). 


Example 38.11: 


i. Let ® bea transformation ®(x) = x + b, and F a linear vector field F(x) = Ax. Then, 


exptF® 1 (x) = e“!(x —b) +b 


Therefore, FeAl (x b) + bl;-9 = A(x — b) = Ax — Ab. Thus, ®yF is an affine vector field. 
ii. If ® isa linear transformation, then d® is also linear, and therefore, PyF = DA®™!,i.e., OyF isa 
linear vector field for any linear field F. 


Theorem 38.5: 


i. If ® is a strong normalizer for a family of vector fields F then, 
@z(LS.(F)) NLie(F) C LS;(F) 
ii. If © is a normalizer for F, then 


®y(LS(F)) N Lie(F) C LS(F) 
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38.7 Applications to Controllability 


The geometric ideas that led to the Lie saturate criterion of controllability provide a beautiful proof of 
controllability of linear systems, demonstrating at the same time that linearity plays an inessential role. 
This proof goes as follows. 

We use the induction on k to show that each controllability space £, = B+ AB+---+A*~'B defined 
in Example 38.8 is contained in the strong Lie saturate of the system. 

Let F denote the family of affine vector fields F,,(x) = Ax + Bu defined by the linear system ae = 
Ax + Bu. For each real number , 0 < X < 1 and each F, in F 


1 
Fy (x) = MAx+ 5 BM) 


belongs to £S;(F) by Theorem 38.4 (ii). Its limit as } — 0 also belongs to £S;(F) since the latter is closed. 
It follows that lim). Fy, = Bu and therefore £; = B is contained in LS,(F). 

Now assume that £,_; C £S;(F). Let b be any element of £,_; and let a be any real number. The 
constant vector field Fy = abisin £,_; for each a. Let By = exp Fy. Then (®y) l= exp F_, and therefore 
both ®,(x) and 51 (x) remain in clA¢(x, < T) for any x € M and T > 0. Therefore, ®,y is a strong 
normalizer for F. According to Theorem 38.5, (®q)#(Fu) C £S;(F) provided that (®)#(F,) € Lie(F). 
Then ((®q)#(Fo))(x) = aByAP__(x) = A(x — ab) because the derivative map ofa translation is equal to 
the identity map. Thus, (®q)4(Fu) belongs to Lie(¥). An analogous argument used in the first step of the 
induction procedure applied to the limit of 4(®« )#(Fo) as \ tends to 0 shows that the constant vector 
field —aAb is contained in £S,(F) for each real ‘number a. But then L; C LS,(F) because the convex 
hull of two vector spaces is the vector space spanned by their sum, ie., 2, = Ce_) + AL,_1. Therefore, 
each L; is in LS,(F). 

When £,_1 = M, the space of all constant vector fields is in the strong Lie saturate and hence 
cl(A#(x, < T))=M for each x € M and T > 0. But then it follows from Theorem 38.2 that M= 
cAr(x,< T) Cc int A¢(x, < T+€) C A¢(x, < T+ €). Therefore, the system is strongly controllable. 

The inductive procedure can also be described pictorially as follows: 


Step 1: Prolong the original system to its closed convex body F. Geometrically F(x) is the translate 
of B to Ax. For each u, Ax + Bu is the line through Ax parallel to Bu, as shown in Figure 38.7a. 
F(x) is the union of all translates of B to points XAx, 0 < d < 1, as shown in Figure 38.7b. 

Step 2: F, contains the vector space B as its edge. Conjugate the original family F by B to obtain a 
prolonged family F given by ae = Ax + Bu+ AByv with both u and v as controls. F2(x) is the 
translate of B + AB to Ax, while the convex body F3 generated by F2 at each point x is the union 


Ax + Bu) 
(a) 


Ax + Bu Bu 


< 
< 


FIGURE 38.7 _ llustration for step 1: (a) F(x) and (b) Fi (x). 
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Ax + Bu + ABv 
(a) 


<N 


x 


FIGURE 38.8 Illustration for step 2: (a) F2(x) and (b) F3(x). 


of all translates of B+ AB to XAx as i ranges in the interval [0, 1]. Figure 38.8 illustrates their 
differences. 

Step 3: Conjugate the original family by the edge B+ AB of F3. The prolonged family is given by 
& = Ax + Bu+ ABv + ABw. 


A repetition of these steps embodied in the induction scheme leads to the saturated system from which 
the controllability properties are evident. 

We now illustrate the importance of the Lie saturate by considering controllability of linear systems 
with bounded controls. Strong controllability is not possible when the constraint set U is compact, because 
each set A¢(x, < T) is compact. It is also known that controllability is not possible whenever the drift 
vector field Ax has an eigenvalue with non-zero real parts. We will now use the geometric framework 
provided by the Lie saturate criterion to obtain affirmative controllability results when the real part of 
the spectrum of A is zero. For simplicity the proof will be given for a particular case only when all the 
eigenvalues of A are zero, i.e., when A is nilpotent. 


Theorem 38.6: 


Suppose that U is a compact neighborhood of the origin in R™, and suppose further that the linear drift 
is nilpotent, i.e., suppose that there is a positive integer p such that AP £0 but APt! =0. Then ae J 


Ax + Bu, u € U is controllable, provided that the rank of (BAB ... A™!B) is equal to dim M. 
Proof. There is no loss of generality in assuming that U is the cube |uj|<¢i=1,...,m. Then the 


reachable set A(0) is a convex neighborhood of the origin in M. Any trajectory x(t) which originates at 
x(0) = 0 is of the form 


t P AKB t 
x(t) = / eA"—5) Bu(s) ds = > ar / (t —s)ku(s) ds 
’ k=0 "°° 


For any u € R”, and any real number 4, there exists T > 0 such that Tail <e for all i=1,...,m. Let 
u(T) = s The corresponding response x(T) is equal to 


Bu ss ABu ee AP-|Bu ie APBu 
TP TPA} piT (p+1)!)’ 
and therefore, limr_, 95 x(T) = si Therefore, the line through A?Bu is contained in the closure of 


A(0). The convex hull of these lines as u ranges over R” is equal to the vector space APB. 
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Take now u(T) = Tr A“. The corresponding trajectory x(T) is given by (8 7s aaa ar Pie + (ett). 
Then, 5 (x(T) _ eet) € clA(0), since the latter is convex. But then limz_,.5 La) — een) = 


nab, A repetition of the previous argument shows that the sum of A?~!B and A?B is contained 
in clA(0). Further repetitions of the same argument show that clA(0) = B+ AB+---+ A?B. The latter 
is equal to M by the rank assumption. 

Since —A is also nilpotent, the above proof is applicable to the time reversed system & = = —Ax — Bu, 
with u € U, to show that its reachable set from the origin is the entire space M. Therefore, any initial 
point x9 can be steered to the origin in some finite time T; using the time-reversed system. But then the 
origin can be steered to any terminal state x; as a consequence of the fact proved above that A(0) = M 
Thus, A(xo) = M for any xo in M and our proof is finished. 


Remark 38.1 


We have implicitly used the Lie saturate criterion to conclude that A(x) = M for all x € M whenever 
clA(x) = M for all x in M. 


38.8 Rotations 


The group of rotations in R? is a natural state space for many mechanical control problems, because the 
kinematics of a rigid body can be described by the movements of an orthonormal frame fixed on the body 
relative to an orthonormal frame fixed in the ambient space. Recall that the rotation group consists of all 
linear transformations R which leave the Euclidean metric (, ) in R? invariant. A Euclidean metric in R? 
is any positive definite scalar product. So if e1, e2, e3 is any orthonormal basis in R? and if x = aan Xiej, 
and y = yy yie; then (x,y) = yy xiyi. R is a rotation if (Rx, Ry) = (x,y) for all x and y in R®. 

Denoting by a, a2,a3 an orthonormal frame fixed on the body, then any motion of the body is 
monitored by the rotation through which the moving frame a1, a2,a3 undergoes relative to the fixed 
frame €1, €2, e3. This rotation, when expressed relative to the basis e1, e2, e3, becomes a 3 x 3 matrix whose 
columns consist of the coordinates of a1, a2, a3 relative to the fixed basis e1, €2, e3. 

The group ofall such matrices whose determinant is equal to 1 is called the special orthogonal group and 
is denoted by SO3(R). SO3(R) is a three-dimensional manifold, which, together with its group structure, 
accounts for a rich geometric base, which needs to be properly understood as a prerequisite for effective 
control of mechanical systems. 

Let us first outline the manifold structure of SO3(R). To begin with, the tangent space of SO3(R) at any 
point Ro consists of all tangent vectors 4 R() |-=<o for curves R(€) in SO3(R) which satisfy R(O) = Ro. The 
tangent space at the group identity I plays a special role and is called the Lie algebra of SO3(R). It consists 
of all matrices A for which e4° € $O3(R). Each such matrix A is antisymmetric because the rotations 
satisfy R-! = R* with R* equal to the transpose, and e~4¢ = (e4¢)* = e4”. Consequently, A* = —A. 

The space of 3 x 3 antisymmetric matrices is a three-dimensional vector space and is denoted by so3(R). 
Each rotation, consisting of orthonormal column vectors, is defined by six orthonormality relations in 
a nine-dimensional group of all 3 x 3 matrices. Therefore, SO3(R) is a three-dimensional manifold, and 
consequently, each tangent space is three-dimensional. But then, the tangent space at I is equal to so3(R). 

Consider now the tangent space at an arbitrary point Ro. For any antisymmetric matrix A each of the 
curves Rj (€) = Roe“€ and Ry(e) = e“€Ro is a curve in SO3(R) which passes through Ro at « = 0. Therefore, 
both am ~ (0) = RoA and #2 (0) = ARp are tangent vectors at Ro. These vectors are different from each 
other foeate of noncommutativity of Ro with A. The first vector is called the left-translation of A by Ro, 
and the second is called the right-translation of A by Ro. It follows that the tangent space at Ro can be 
described by either left- or right-translations of so3(R). 
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Denote by Aj, Az, A3 the standard basis of so3(R), 


0 0 O 0 oO 1 0 -1 0O 
A;=|]0 0 -1 Ary=|0 0 0 Az3=]1 0 O 
0 1 0 -1 0 0 0 oO O 
Since Aje; = 0, it follows that each e4 is a rotation about the axis containing e;,i= 1,2,3. For any 
0 —a2 a2 ay 
antisymmetric matrix A= | 4a 0 —a, | we will use A to denote the column vector | a> |. A is 
—a2 a, 0 a3 


the coordinate vector of A relative to the standard basis. 

Any antisymmetric matrix A induces vector fields on SO3(R). The first vector field is given by F)(R) = 
RA, and the second is F,(R) = AR. F; is called the left-invariant vector field induced by A because its 
tangent vector at R is a left translation by R of its tangent vector at the group identity. Similar explanations 
apply to right-invariant vector fields F,. We will use A; to denote the left-invariant vector field whose 
tangent at I is equal to A, ie., A; (R) = RA. Similarly, A; denotes the right-invariant field A,(R) = AR. 

Then (A; Yb (A>); and (A3); is a basis of left-invariant vector fields which span each tangent space and 
(Ai) rs (Ad), and (A3), is a basis of right-invariant vector fields with the same property. 

Any differentiable curve R(t) in SO3(R) defines a curve of tangent vectors ak at R(t), which can be 


expressed by either right or left basis. Let am — Siar w;(t)(Az),(R) = Se Q;(t)(A;))(R(t)) denote the 


o1(t) Qi (t) 
corresponding coordinates of Rm Vectors w(t) = | w2(t) }, and Q(t) = | Q2(t) | are called the angular 
o3(t) Q3(t) 


velocities of R(t). In analogy with the kinematics of a rigid body, the first angular velocity is called the 
(absolute) angular velocity, while the second is called the body angular velocity. The above differential 
equations can be rewritten as 


0 —w3(t) w(t) dR(t) 0 —Q3(t) Q2(t) 
w3(t) 0 —@ (t) | RU) = oe Ta R(t) | Q3(¢) 0 —Q,(t) 
—@2(t) @1(£) 0 —Q2(t) = Qy(t) 0 


It can be shown that Q(t) = R7!(t)w(t). 

Any differentiable curve R(t) whose angular velocity is constant is a solution curve of an invariant 
vector field. If w(t) is constant, then oH = AR with A =, and if Q(t) is constant then oR = RA with 
A= Q. In the first case, R(t) = e4*Ro while in the second case R(t) = Roe. 

It can be shown that the Lie bracket of a right (respectively, left) invariant vector fields is a right 
(respectively, left) invariant vector field, with (Ay, By (R) = R(BA — AB) and [A,, B,|(R) = (AB— BA)R. 

It is easy to verify that the commutator AB — BA can also be expressed in terms of the cross-product 
of A and B in R? as follows: let [A, B], = AB— BA and [A, B]; = BA — AB. Then, [A, B], = A x B, while 
[A, Bl, = B x A. 

Except for the cross-product correspondence, all of these concepts extend to the rotation group SO,(R) 
of R”, and its mn) -dimensional Lie algebra so,(R) of n x n antisymmetric matrices. 


38.9 Controllability in SO,(R) 


A unit sphere which rolls on a horizontal plane without slipping and without spinning along the axis 
perpendicular to the point of contact, can be described by the following equations: 


dx} 
—_—= t 
di uy (t) 
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dx2 
—_~= t 
dt u2(t) 
0 0 al 
dR(t) 
a 0 R(t 
7 u2 | R(t) 


x1(t) and x(t) are the coordinates of the center of the sphere (x3 = 1), and R(t) is the orientation of the 
=e 
sphere relative to an absolute frame e1, e2, e3. The angular velocity w(t) = | wu, | of the sphere is always 
0 
orthogonal to the velocity of its center. 

The rotational kinematics of the sphere may be viewed as a left-invariant control system on SO3(R) with 
two controls u; and u2. This control system has no drift, and therefore, according to a well-known theorem 
of geometric control theory, the system is strongly controllable whenever the Lie algebra generated by 
the controlling vector fields is equal to the Lie algebra of the group (in this case SO3(R)). It follows that 
the controlling vector fields are Fj(R) = A2R and F(R) = —A,R corresponding to uw; = 1, uz = 0 and 
uz = 0, u; = 1. The rotational part is strongly controllable since [F,, F2](R) = A3R. It can also be shown 
that the overall system in R? x SO3(R) is strongly controllable because the Lie algebra generated by the 
controlling vector fields is equal to R? x so3(R). 

There is a simple argument showing that any states in R? x SO3(R) can be transferred to each other 
by two switches in controls. Note first that for any angular velocity A the corresponding rotation e4 is 
the rotation about A through the angle ||Al|. Figure 38.9 shows that any rotation can be achieved by one 
switch in controls (two angular velocities w; and w2). 

The proof begins with the observation that each unit circle in the e), e2 plane centered at the origin 
has a line w in common with the circle in the a), az plane also centered at the origin. w is in the plane 
«3 = 0 as shown in the picture. The first move consists of rotating about w so that a3 coincides with —e). 
Then rotate through x radians along the midpoint of the arc between az and e2. These two moves rotate 
any frame a), 42,43 into the standard frame. The remaining moves are used to roll for the position of 
the point of contact along a line segment whose length is an integral multiple of 2. Such moves do not 
alter the orientation of the ball. Any two points in the plane can be joined along the sides of an isosceles 
triangle with equal sides equal to 21m, as shown in Figure 38.10. 

The reader may note the similarity of this argument with the one used to show that any rotation in R? 
may be achieved by the rotations through the Euler angles , 0, and p. 

This exposition ends with a controllability theorem whose proof also relies on the Lie saturate. 
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FIGURE 38.9 Rotational kinematics. 
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FIGURE 38.10 Translational kinematics. 


Theorem 38.7: 


Suppose that F is any family of right (or left) invariant vector fields on SO,(R) (or any other compact 
Lie group G). Then F is controllable if and only if Lie(¥), evaluated at I, is equal to the Lie algebra of 
SO;,(R) (or G). 


The proof consists in showing that —F is contained in the Lie saturate of . Therefore, the vector 
span of F is contained in £ S(F) by the convexity property of £S(F). But then the Lie algebra of F is 
contained in the Lie saturate and, hence, must be equal to it. The controllability result then follows from 
the Lie saturate criterion. So, the proof will be complete once we showed that (exp —tF)(R) € clAF(R) 
for any tf > 0 and any F € F. Let F(R) = AR. Then, (exp —tF)(R) = e ‘AR, and therefore (exp —tF)(R) 
belongs to clA¥(R) if and only if e~™ belongs to the closure of the reachable set from the group identity. 

S0,(R) is a compact group and therefore there exists a sequence of times f, tending to oo such that 
lim e’n4 exists. Let limy_5o e!"4 = Ro. Then, Ro ' = limy soo e~ 4". If necessary, choose a subsequence so 
that ty41 — ty also tends to oo. Then, I = Ry Ro = (limps 09 e744) (lim ef +14) = limy 99 ebm A, 

The preceding argument shows that e’4 comes arbitrarily close to the identity for large values of time. 
Then, 


eA — eFA( Jim eli WA) = Jim ellfnt1—fn)—A 
n—> oo n> Co 


Since tn41 — th > ©, (tn41 — ty) — t > 0 for large values of n and therefore eA EclAr(I). The proof is 
now finished. 

Theorem 38.7 might be used to show that the orientation of a rigid body may be controlled by any 
number of gyros situated on the body as long as the Lie algebra generated by their angular velocities has 
full rank. 
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Further Reading 


The proofs of all theorems quoted in this paper can be found in the forthcoming book titled Geometric 
Control Theory by V. Jurdjevic, (to appear in Studies in Advanced Mathematics, Cambridge University 
Press.) The material for this article is taken out of the first part of the book dealing with the reachable sets 
of Lie determined systems (which includes analytic systems). 

The reader may also find some of this material in the following publications: 


1. Jurdjevic, V. and Kupka, I.A., Polynomial control system, Math Ann., 361-368, 1985. 

2. Jurdjevic, V. and Sussmann, H.J., Control systems on Lie groups, J. Diff Eqs., 12, 313-329, 1972. 

3. Sussmann, H.J. and Jurdjevic, V., Controllability of non-linear systems, 1. Diff, Egs., 12, 95-116, 
1972. 


Convexification of control systems is also known as the relaxation of controls in the early literature on 
control. See for instance, 


4. Hermes, H. and LaSalle, J.P., Functional Analysis and Time Optimal Control, Academic Press, New 
York, 1969. 

5. Warga, T., Optimal Control of Differential and Functional Equations, Academic Press, New York, 
1972. 


For other applications of Lie theory to control systems the reader may consult Geometric Methods in 
Systems Theory, Proceedings of the NATO Advanced Study Series, Editors: R. Brockett and D. Q. Mayne, 
Reidel Publishing, 1973. 
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39.1 Introduction 


In this chapter we present the asymptotic methods of averaging and singular perturbation. Suppose we 
are given the state equation x =f (t,x,¢), where ¢ is a “small” positive parameter, and, under certain 
conditions, the equation has an exact solution x(t,¢). Equations of this type are encountered in many 
applications. The goal of an asymptotic method is to obtain an approximate solution x(t, ¢) such that 
the approximation error x(t, ¢) — x(t, €) is small, in some norm, for small ¢ and the approximate solution 
x(t, €) is expressed in terms of equations simpler than the original equation. The practical significance of 
asymptotic methods is in revealing underlying multiple timescale structures inherent in many practical 
problems. Quite often the solution of the state equation exhibits the phenomenon that some variables 
move in time faster than other variables, leading to the classification of variables as “slow” and “fast.” 
Both the averaging and singular perturbation methods deal with the interaction of slow and fast variables. 


39.2 Asymptotic Methods 


We start by a brief description of the perturbation method that seeks an approximate solution as a finite 
Taylor expansion of the exact solution. Then, we introduce the averaging method in its simplest form, 
which is sometimes called “periodic averaging” since the right-hand side function is periodic in time. 
Next, we introduce the singular perturbation model and give its two timescale properties. Finally, we 
show how to improve the accuracy of the reduced model of a singularly perturbed system. 


39.2.1 The Perturbation Method 
Consider the system 
x=f(tx,8), x(t)=y7 (39.1) 


39-1 
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where f is sufficiently smooth in its arguments in the domain of interest, and ¢ is a positive parameter. 
The solution of Equation 39.1 depends on the parameter ¢, a point that we shall emphasize by writing the 
solution as x(t, €). The goal of the perturbation method is to exploit the “smallness” of the perturbation 
parameter ¢ to construct approximate solutions that will be valid for sufficiently small ¢. The simplest 
approximation results by setting ¢ = 0 in Equation 39.1 to obtain the nominal or unperturbed problem: 


x=f(t,x,0), x(t.) =n (39.2) 


Suppose this problem has a unique solution xo(t) defined on [to, t;]. By continuous dependence of the 
solutions of differential equations on parameters, we know that, for sufficiently small ¢, the system 
equation 39.1 has a unique solution x(t, ¢), defined on [fo, t)], such that 


||x(t, €) —xo(t)|| < ke, Ve<e,, Vt €[to, th] 


for some k > 0 and ¢, > 0. In this case, we say that the error is of the order O(g) and write x(t, €) — xo(t) = 
O(e). This order of magnitude notation will be used frequently. It is defined as follows. 


Definition 39.1: 


81 (€) = O(82(e)) if there exist positive constants k and c such that 
ISi(e)| < AlSa(e)], Vel <e 
Higher-order approximations for the solution of Equation 39.1 can be obtained in a straightforward 
manner. We construct a finite Taylor series 


N-1 
x(t, €) = x9(t) + De xp(t)e* + & r(t, €) (39.3) 
k=1 


Substitution of Equation 39.3 into Equation 39.1 yields 


N-1 
Y- dae +eNF(t,6) =f lt x(tse),€) = h(te) (39.4) 
k=0 


where the coefficients of the Taylor series of h(t, ¢) are functions of the coefficients of the Taylor series 
of x(t,€). Since the equation holds for all sufficiently small ¢, it must hold as an identity in e. Hence, 
coefficients of like powers of ¢ must be equal. Matching those coefficients we can derive the equations 
that must be satisfied by xo, x;, and so on. The zeroth-order term ho(t) is given by ho(t) = f(t, xo(t), 0). 
Hence, matching coefficients of &° in Equation 39.4, we determine that xo(t) satisfies 


xo =f (t,x0,0), xo(to) =n 


which, not surprisingly, is the unperturbed problem of Equation 39.2. The first-order term h(t) is 
given by 


0 0 
hy(t) = 2 (t,x0(0).0) x(t) + Z(t,x0(0,0) 
ba Oe 


Matching coefficients of ¢ in Equation 39.4 we find that x(t) satisfies 


x, = A(t)x1 + g(t, xo(t)), x1 (to) = 0 (39.5) 
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where 


) 6) 
A(t) = 2 (200 0), gilt, xo(t)) = 2 x00, 0) 


This linear equation has a unique solution defined on [to, 1]. This process can be continued to derive the 
equations satisfied by x2, x3, and so on. By straightforward error analysis, it can be established that 


N-1 
x(t,e)— ) > xx(t)e* = O(e%) (39.6) 


k=0 


The O(eN) error bound in Equation 39.6 is valid only on finite time intervals [fo, t)]. It does not hold 
on intervals like [fo, T’/e] nor on the infinite time interval [to, 00). The reason is that the constant k in the 
bound ke depends on t; and may grow unbounded as ty increases. The error bound in Equation 39.6 can 
be extended to the infinite time interval [to, 00) if some additional conditions are added to ensure stability 
of the solution of the nominal system of Equation 39.2. In particular, suppose that Equation 39.2 has an 
exponentially stable equilibrium point p*, then Equation 39.6 holds on the infinite time interval [f0, 00), 
provided 1 is sufficiently close to p*. We recall that a solution x(t) of a state equation is exponentially 
stable if for x(0) sufficiently close to x(0), the inequality 


I|x(t) — x(£)|| < kllx(0) — x(0)|| exp(—yt), Wt>0 


is satisfied with some positive constants k and y. This definition applies whether x is an equilibrium point 
or a periodic solution. For a time-invariant system x = f (x), an equilibrium point p* is exponentially stable 
if the Jacobian matrix [Of /Ox], evaluated at x = p*, has eigenvalues with negative real parts. 


39.2.2 Averaging 


The averaging method applies to a system of the form 
x = ef (t,x, 8) (39.7) 
where ¢ is a small positive constant and f(t, x, €) is T-periodic in t, uniformly in (x, ¢); that is, 
f(t,x,e) =f(t+ T,x,8) 


for all x and e. We assume that f is sufficiently smooth in its arguments over the domain of interest. The 
method approximates the solution of Equation 39.7 by the solution of the time-invariant average system 


x = tfay(x) (39.8) 
where 


T 
for) = = [ Hae (39.9) 


The intuition behind this approximation can be seen as follows. When ¢ is small, the solution x will 
vary “slowly” with ¢ relative to the periodic fluctuation of f(t,x,¢). Therefore, x will be determined 
predominantly by the average of f. This intuition has its roots in linear system theory where we know 
that if the bandwidth of the system is much smaller than the bandwidth of the input, then the system will 
act as a low-pass filter that rejects the high-frequency component of the input. 

The basic problem in the averaging method is to determine in what sense the behavior of the time- 
invariant system of Equation 39.8 approximates the behavior of the time-varying system of Equation 39.7. 
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We shall address this problem by showing, via a change of variables, that the system of Equation 39.7 can 
be represented as a perturbation of the system of Equation 39.8. Define 


t 
u(t, x) = i Uf (t, x, 0) — fay(x)] dt (39.10) 
0 


Since f (t,x, 0) — fay(x) is T-periodic in t with zero mean, the function u(t, x) is T-periodic in t. It can be 
also shown that Ou/Ot and Ou/Ox are T-periodic in t. The change of variables 


x=y+eu(t,y) (39.11) 
transforms Equation 39.7 into the form 


¥ =efay(y) +e°q(t,y, 8) (39.12) 


where q(t, y, €) is T-periodic in tf. This equation is a perturbation of the average system of Equation 39.8. 
It can be represented as a standard perturbation problem by changing the time variable from t to s = et. 
In the s timescale the equation is given by 


dy s 
aE = fav(y) + €q (<.». e) (39.13) 


where q(s/e, y, €) is ¢T -periodic in s. 
The problem has now been reduced to the perturbation problem we studied in the previous section. If, 
for a given initial state x(0) = n, the average system 


dy - _ 
a =faly), yO)=n 


has a unique solution y(s) defined on [0, b], then for sufficiently small ¢ the perturbed system of Equation 
39.13 will have a unique solution defined for all s € [0, b] and the two solutions will be O(¢) close. Since 
t=s/e and x — y = O(e), by Equation 39.11, the solution of the average system of Equation 39.8 provides 
an O(&) approximation for the solution of Equation 39.7 over the time interval [0, b/] in the t timescale. If 
the average system of Equation 39.8 has an exponentially stable equilibrium point p* and Q is a compact 
subset of its region of attraction, then for all initial states in Q, the O() approximation will be valid for 
allt > 0. 

Investigation of Equation 39.13 reveals another interesting relationship between Equations 39.7 and 
39.8. If Equation 39.8 has an exponentially stable equilibrium point p*, then Equation 39.7 has a unique 
exponentially stable T-periodic solution in an O(e) neighborhood of p*. 

The averaging method can be extended to systems where the right-hand side of Equation 39.7 is not 
periodic in t, if an average of f (t, x, 0) can be defined by the limit 


1 t+T 
fav(x) = jim T / f (t,x, 0) dt 


A simple example is the case when f(t, x, 0) = fi(t, x) + fo(t, x), where f; is periodic in t while f) satisfies 
f(t, x)|| < keV! over the domain of interest. 


39.2.3 Singular Perturbation 


While the perturbation method of Section 39.2.1 applies to state equations that depend smoothly ona small 

parameter ¢, in this section we face a perturbation problem characterized by discontinuous dependence 
of system properties on the perturbation parameter ¢. We shall study the singular perturbation model 

x =f (t,x, Z,€) (39.14) 

eZ = g(t, x, Z,€) (39.15) 


where setting ¢ = 0 causes a fundamental and abrupt change in the dynamic properties of the system, as 
the differential equation ez = g degenerates into the algebraic or transcendental equation 0 = g(t, x, z, 0). 
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The essence of the theory described in this section is that the discontinuity of solutions caused by singular 
perturbations can be avoided if analyzed in separate timescales. This multi timescale approach is a 
fundamental characteristic of the singular perturbation method. 

Consider the singularly perturbed system of Equations 39.14 and 39.15 where x € R” and z € R”. We 
assume that the functions f and g are sufficiently smooth in the domain of interest. When we set ¢ = 0 
in Equation 39.15, the dimension of the state equation reduces from n+ m to n because the differential 
equation 39.15 degenerates into the equation 


0 = g(t, x, z, 0) (39.16) 


We shall say that the model of Equations 39.14 and 39.15 is in the standard form if and only if Equation 
39.16 has k > 1 isolated real solutions 


z=hj(t,x), i=1,2,...,k (39.17) 


for each (t,x) in the domain of interest. This assumption ensures that a well-defined n-dimensional 
reduced model will correspond to each solution of Equation 39.16. To obtain the ith reduced model, we 
substitute Equation 39.17 into Equation 39.14, at ¢ = 0, to obtain 


x =f (t,x, h(t, x), 0) (39.18) 


where we have dropped the subscript i from h. It is usually clear from the context in which the solution 
of Equation 39.17 is being used. This model is called quasi-steady-state model because z, whose velocity 
z= g/e can be large when ¢ is small and g £0, may rapidly converge to a solution of Equation 39.16 
which is the equilibrium of Equation 39.15. The model of Equation 39.18 is also known as the slow model 
or the reduced model. 

Singular perturbations cause a two timescale behavior characterized by the presence of slow and fast 
transients in the system’s response. Loosely speaking, the slow response is approximated by the reduced 
model of Equation 39.18, while the discrepancy between the response of the reduced model (Equation 
39.18) and that of the full model of Equations 39.14 and 39.15 is the fast transient. To see this point, let 
us consider the problem of solving the state equation 


x=f(t,x,z,€), x(to)=€E (39.19) 
ez = g(t,x,z,£), Z(t) =n (39.20) 


Let x(t,¢) and z(t,e) denote the solution of the full problem of Equations 39.19 and 39.20. When we 
define the corresponding problem for the reduced model of Equation 39.18, we can only specify n initial 
conditions since the model is of nth order. Naturally we retain the initial state for x, to obtain the reduced 
problem 

x=f(t,x,h(t,x),0), x(to) =& (39.21) 


Denote the solution of Equation 39.21 by x(t). Since the variable z has been excluded from the reduced 
model and substituted by its “quasi-steady-state” h(t, x), the only information we can obtain about z by 


solving Equation 39.21 is to compute z(t) ae h(t, x(t)), which describes the quasi-steady-state behavior of 
zwhen x = x. By contrast to the original variable z, starting at to from a prescribed n, the quasi-steady-state 
Z is not free to start from a prescribed value, and there may be a large discrepancy between its initial value 
Z(to) = h(to, &) and the prescribed initial state n. Thus z(t) cannot be a uniform approximation of z(t, €). 
The best we can expect is that the estimate z(t, €) — z(t) = O(e) will hold on an interval excluding fo, that is, 
for t € [ty, t)] where ty > fo. On the other hand, it is reasonable to expect the estimate x(t, €) — x(t) = O(e) 
to hold uniformly for all ¢ € [t, t] since x(to, €) = x(to). 

If the error z(t,¢)— Z(t) is indeed O(e) over [t,,t)], then it must be true that during the initial 
(“boundary layer”) interval [f9, t,] the variable z approaches z. Let us remember that the speed of z 
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can be large since z = g/g. In fact, having set ¢ = 0 in Equation 39.15, we have made the transient of z 
instantaneous whenever g # 0. To analyze the behavior of z in the boundary layer, we set y = z — h(t, x), 
to shift the quasi-steady-state of z to the origin, and change the timescale from t to t = (t — fo)/s. The 
new time variable t is “stretched”; that is, if tends to zero, t tends to infinity even for finite t only slightly 
larger than fo by a fixed (independent of ¢) difference. In the t timescale, y satisfies the equation 


d h h 
y g(t. x, y+ A(t, x), €) — aC xy + h(t, x),€) — Oxy + h(t, x), e)f (t,x, y + h(t, x), ©) 
x 


dt 
(39.22) 
with y(0) = y — A(to, &). The variables t and x in the foregoing equation will be slowly varying since, in 
the t timescale, they are given by 


t=totet, x=x(tp + 6t,&) 


Setting « = 0 freezes these variables at their initial values and reduces Equation 39.22 to the time-invariant 
system 


d 

= gltorsy + ho, 8).0), yO) = 0(0) — h(to.8) (39.23) 
which has equilibrium at y = 0. The frozen parameters (fo, &9) in Equation 39.23 depend on the given 
initial time and initial state. In our investigation of the stability of the origin of Equation 39.23 we should 
allow the frozen parameters to take any values in the region of the slowly varying parameters (tf, x). We 


rewrite Equation 39.23 as 


d 
oa = g(t,x,y + h(t, x), 0) (39.24) 
T 
where (tf, x) are treated as fixed parameters. We shall refer to Equation 39.24 as the boundary-layer model 
or the boundary-layer system. The crucial stability property we need for the boundary-layer system is 
exponential stability of its origin, uniformly in the frozen parameters. The following definition states this 


property precisely. 


Definition 39.2: 


The equilibrium y = 0 of the boundary-layer system of Equation 39.24 is exponentially stable uniformly in 
(t,x) if there exist positive constants k and y and a compact set Q, containing the origin, such that for each 
y(0) € Q the solution of Equation 39.24 satisfies the inequality 

Iy(D)Il < Klly)|l exp(—yv),_ Vt2 0 (39.25) 


for all (t,x) in the domain of interest. 


If the Jacobian matrix [Og/Oy] satisfies the eigenvalue condition 


Re E { e. x, h(t, x), 0] <-c<0 (39.26) 
oy 


for all (t,x) in the domain of interest, then there exist k, y, and Q for which the inequality of Equation 
39.25 is satisfied. 

Under the boundary-layer stability condition, the fundamental result of singular perturbation, known 
as Tikhonov’s theorem, states that if the reduced problem of Equation 39.21 has a unique solution x(t), 
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defined on [to, ti], and n — h(to, &) € Q, then for sufficiently small ¢, the full problem of Equations 39.19 
and 39.20 has a unique solution (x(t, €), z(t, €)) defined on [to, ti], and 


x(t, €) — x(t) = O(e) (39.27) 


z(t, €) — h(t, x(t)) — p(t — to) /e) = Ole) (39.28) 


hold uniformly for t € [to, ti], where (t) is the solution of the boundary-layer model of Equation 39.23. 
Moreover, given any tp > to, 
z(t, 8) — h(t, x(t)) = O(e) (39.29) 


holds uniformly for t € [tp, t)]. An infinite time version of this result holds when the reduced system of 
Equation 39.18 has an exponentially stable equilibrium point. 


39.2.4 Model Order Reduction 


The singular perturbation method provides a systematic procedure to obtain a reduced-order model of 
a two timescale system by neglecting its fast dynamics. In this section we show how we can improve the 
accuracy of the reduced model. We consider a time-invariant special case of Equations 39.14 and 39.15, 
namely, 


n= F(z) (39.30) 
eZ = g(x, z) (39.31) 
in which f and g do not depend on ¢. Let z = h(x) be an isolated solution of the equation 0 = g(x, z) and 
suppose the assumptions of Tikhonov’s theorem are satisfied with a nonsingular Jacobian matrix [Og /0z]. 


The equation z = h(x) describes an n-dimensional manifold in the the (n + m)-dimensional state space 
of x and z. It is an invariant manifold for the system 


x =f (x, Zz) 
0 = g(x, Z) 


since a trajectory starting in the manifold z = h(x) stays in the manifold for all future (or past) time. The 
existence of an invariant manifold will carry over to the case ¢ > 0 when ¢ is sufficiently small. We seek 
the manifold equation in the form 


Pees (39.32) 


where ¢ is sufficiently smooth. Equation 39.32 defines an e-dependent n-dimensional manifold. For 
z= (x, €) to be an invariant manifold for the system of Equations 39.30 and 39.31, it must be true that 


aac (x, €)) = g(x, o(x, €)) (39.33) 


for all x in the domain of interest and all ¢ € [0, ¢9]. It can be shown that, for sufficiently small ¢, there is 
a function ¢ that satisfies Equation 39.33 and (x, 0) = h(x). The reduced-order model 


k= f(x, o(x,8)) (39.34) 


is an exact slow model that describes the motion of the system on the invariant manifold z = (x, ). The 
reduced model of the previous section, namely, 


x =f (x, h(x)) 


39-8 Control System Advanced Methods 


is an approximation of the model of Equation 39.34. We can improve the approximation of the model by 
seeking the solution of Equation 39.33 in the power-series form 


(x, €) = bo(x) + 81 (x) + e7o(x) +++ (39.35) 


where ¢o(x) = h(x). The series of Equation 39.35 is substituted into Equation 39.33 and the terms oo, 
$1,--+ are calculated by matching the coefficients of like powers of ¢. The more terms we include in the 
approximation of ¢, the more accurate the reduced model is. 


39.3 Examples 


We give four examples to illustrate the foregoing discussion. Example 39.1 illustrates the averaging 
method. Example 39.2 shows how the averaging method can be used to detect the existence of limit cycles 
in weakly nonlinear second-order systems. Example 39.3 illustrates the singular perturbation method. 
Example 39.4 shows how the accuracy of the reduced model can be improved. 


Example 39.1: 


Consider the scalar system 
x =e(x sin? t — 0.5x2) = ef (t,x) 


The function f(t, x) is 1-periodic in t. The average function fay (x) is given by 
1 IU 
fay (x) = — i (x sin? t— 0.5x2) dt =05(x — x?) 
wT JO 


The average system 

xX = 0.5e(x — x) 
has an exponentially stable equilibrium point at x = 1 since [dfay /dx](1) = —0.5. Hence, for suffi- 
ciently small ¢, the original system has an exponentially stable m-periodic solution in an O(e) neigh- 
borhood of x = 1. Moreover, for initial states sufficiently near x = 1, solving the average system with 


the same initial state as the original system yields the approximation 
X(t, €) — Xqv(et) = O(c), VWt>O 


Figure 39.1 shows the solutions of the original and average systems for the initial state x(0) = 0.5 and 
€= 0.2. 


12 T T T T T T T 


State variable 


04 ! ! ! ! 
0 5 10 15 20 25 30 35 40 


Time 


FIGURE 39.1 The exact (solid) and average (dashed) solutions of Example 39.1 with ¢ = 0.2. 
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Example 39.2: 


Consider the Van der Pol equation 


XQ = —X1 + €Xx2(1 — x?) 
Representing the system in the polar coordinates 
X}=rsind, xX2=rcosd 
it can be shown that 


+= ercos? o(1—r? sin? 4) (39.36) 
b=1-sesindcos (1 —r sin? >) (39.37) 


Divide Equation 39.36 by Equation 39.37 to obtain 


a ercos*(1—r sin?) def 
: me 39.38 
dp 1—esindcos¢ (1 — r2 sin? ) ef (, 1, €) ( 


If we view ¢ as the independent variable, then Equation 39.38 takes the form of Equation 39.7, where 
f(,r, €) is 2-periodic in ). The average system 


ae ah pee (39.39) 
do =€ 5 = 3 ‘ 
has an exponentially stable equilibrium point at r = 2 since [dfgy /dr](2) = —1. Therefore, for suffi- 


ciently small «, Equation 39 38 has an exponentially stable 21-periodic solution r = R(, €) in an O(e) 
neighborhood of r = 2. Substitution of r = R(4, €) into Equation 39.37 yields 


6 =1—esin cos ¢ (1 — R2(6, €) sin? $) (39.40) 


Let *(t, e) be the solution of Equation 39.40 that starts at *(0, ¢) = 0. It can be argued that there is 
T(e) = 2x + O(e) such that 


o*(t+T(e),e) = 204+ o*(t,e), Vt>O0 


Hence, 
R(p* (t + T(e), €), €) = R(2x + O* (Ct, €), €) = R(b*(C, ), €) 


which shows that the Van der Pol equation has a stable limit cycle in an O(¢) neighborhood of r = 2. 
The period of oscillation is O(e) close to 2x. 


Example 39.3: 
Consider the singular perturbation problem 


x=x2(14+t)/z, x(0)=1 
ez=—[Z+(1+t)x]z[z—-(1+ 0], z(0)=y 
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Setting e = 0 yields the equation 

0=—[z+(14+ dx] z[z-(14+ B)] 
which has three isolated solutions 

z=—(1+t)x; z=0; z=1+t 


in the region {t >0 and x > k}, where 0<k <1. Consider first the solution z = —(1+ t)x. The 
boundary-layer model of Equation 39.24 is 


Using the Lyapunov function V(y) = hy, it can be verified that the boundary-layer stability condi- 
tion is satisfied for |y(0)| < p < (1+ t)x. The reduced problem 


has the unique solution x(t) = exp(—t) for all t > 0. The boundary-layer problem with t = Oandx = 1 
is 
d 
Y =-yy-Wy-2), yo)=n+ 
dt 
and has a unique exponentially decaying solution y(t) for 7 < 0. Consider next the solution z = 0. 
The boundary-layer model of Equation 39.24 is 


dy 


7 =-[y+(14+t)x]yy-(4+0] 
a 


By sketching the right-hand side function, it can be shown that the origin is unstable. Hence, 
Tikhonov's theorem does not apply to this case. Finally, the boundary-layer model for the solution 
z=1+tis 
dy 
dt 
Similar to the first case, it can be shown that the origin is exponentially stable uniformly in (t, x). The 
reduced problem 


=-[y+(1+t)+(14+tx]y+(1+0ly 


k= x2, x(0) =1 


has the unique solution x(t) = 1/(1 — t) for all t € [0, 1). Notice that x(t) has a finite escape time at 
t = 1. However, Tikhonov's theorem still holds for t € [0, t;] witht; < 1. The boundary-layer problem, 
with t =O andx = 1, 

dy a 


at =—-(yt+2)y+1)y, y(0)=yn-1 
T 


has a unique exponentially decaying solution y(t) for y > 0. Among the three solutions of Equation 
39.16, only two solutions, h = —(1+4 t)x andh= 1 +t, give rise to valid reduced models. Tikhonov’s 
theorem applies to the solution h= —(1+t)x if 7 <0 and to the solution h=1+t if n>0. 
Figure 39.2 shows the exact and approximate solutions of x and z for y = —2 and e = 0.3. The z 
solution exhibits a two timescale behavior. It starts with a fast transient from 7 to the reduced solu- 
tion Z(t). After the decay of this transient, it remains close to Z(t). The approximation Z(t) + y(t/e) is 
valid for all t € [0, 1], while the approximation Z(t) is valid only after the boundary-layer period. As 
for the slow variable x, the approximation x(t) is valid for all t € [0, 1]. 


Example 39.4: 


Consider the singularly perturbed system 


x = xX2 
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State variables 


Time 


FIGURE 39.2 The exact (solid), reduced (dashed), and approximate (dotted) solutions of Example 39.3 
with e = 0.3. 


X= xy + (1 —x2)z 


&Z = X72 —-Z 
The manifold condition of Equation 39.33 takes the form 


Oo Oo 24) | vs 
| Sra + Fe x +(1 196) | =% 


Seeking the solution ¢ in the power series of Equation 39.35 and matching the coefficients of like 
powers of ¢, we see that dg = X2, which results in the reduced model 


xy =xX2 


X= —xX, + (1 —x?)xo 


The next term $1 is given by 6; =x; —(1 —x?)x. Approximating » by $9 + €61 results in the 
corrected reduced model 
xy =xX2 


X= —x,+(1 — x? )[x2 +(x; -—(1 —x?)x9)] 


Figure 39.3 shows the improvement in approximating x(t) as we go from the reduced to the corrected 
reduced models. The calculations are done with the initial conditions xj (0) = x(0) = 1 and z(0) = 0. 


39.4 Defining Terms 


Average system: A time-invariant system obtained by averaging the fast periodic right-hand side 
function of a time-varying system. 

Boundary-layer model: A reduced order model that describes the motion of the fast variables of a 
singularly perturbed system in a fast timescale where the slow variables are treated as constant 
parameters. 

Equilibrium point: A constant solution of «= f(t,x). For the time-invariant system x = f(x), the 
equilibrium points are the real solutions of the equation 0 = f(x). 

Exponentially stable solution: A solution (e.g., equilibrium point or periodic solution) is exponentially 
stable if other solutions in its neighborhood converge to it faster than an exponentially decaying 
function. 
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State variable 


0 5 10 15 20 25 30 


Time 


FIGURE 39.3. The exact (solid), reduced (dashed), and corrected reduced (dotted) solutions of Exam- 
ple 39.4 with e=0.1. 


Linearization: Approximation of the nonlinear state equation in the vicinity of a nominal solution 
by a linear state equation, obtained by dropping second- and higher-order terms of the Taylor 
expansion (about the nominal solution) of the right-hand side function. 

Periodic solution (orbit): A periodic solution x(t) of x = f(t, x) satisfies the condition x(t + T) = x(t), 
for all t > 0 for some positive constant T. The image of a periodic solution in the state space is a 
closed orbit. 

Reduced Model: A reduced-order model that describes the motion of the slow variables of a singularly 
perturbed system. The model is obtained by setting ¢ = 0 and eliminating the fast variables. 

Region of attraction: A domain containing an asymptotically stable equilibrium point such that all 
trajectories starting in the domain converge to the point. 

Stable limit cycle: An isolated periodic orbit such that all trajectories in its neighborhood asymptotically 
converge to it. 

Standard singularly perturbed model: A singularly perturbed model where upon setting ¢ = 0, the 
degenerate equation has one or more isolated solutions. 

Time-invariant system: A state equation where the right-hand side function is independent of the time 


variable. 
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For Further Information 


Our presentation of the asymptotic methods is based on the textbook by Khalil [1]. For further information 
on this topic, the reader is referred to Chapters 10 and 11 of Khalil’s book. Chapter 10 covers the 
perturbation method and averaging, and Chapter 11 covers the singular perturbation method. Proofs of 
the results stated here are given in the book. 

The discussion of model order reduction is based on Chapter 1 of Kokotovic et al. [2]. This book gives a 
broader view of the use of singular perturbation methods in systems and control. Modeling two timescale 
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systems in the singularly perturbed form is discussed in Chapters 1 and 2 of Kokotovic [2] and Chapter 11 
of Khalil [1]. 

For a broader view of the averaging method, the reader may consult Sanders, J.A., Verhulst, F. and 
Murdock, J. 2007. Averaging Methods in Nonlinear Dynamical Systems, 2nd Edn., Springer, Berlin. A 
unified treatment of singular perturbation and averaging is given in Teel, A.R., Moreau, L. and Nesic, D. 
2003. A unified framework for input-to-state stability in systems with two timescales, IEEE Transactions 
on Automatic Control, vol. 48, no. 9, 1526-1544. 
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40.1.1 Some Simple Examples 


Consider the linear system 


x(t) = Ax(t) + Bu(t), 
y(t) = Cx(t) 


Its solution may be written in the form 


xER", uEeR, x(0)=x9 


t 
y(t) = CeA*xg + i CeA—9 Bu(t) dt. (40.1) 


0 


On the other hand, the scalar time-varying linear system 


fee =a(t)x+ b(t)u(t), x,y,uweER, x(0)= x0 


y(t) = c(t)x(t) 


has a solution 


t t t 
y(t) = c(t)eo a(v)dt) +f c(t)et 4(0)40) b(t) u(t) dt: (40.2) 
0 


40-1 
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Now consider the system 


fee = (Dx(t)+ B)u(t), x€R’, y,ueR, x(0) = (0, 0) 
y(t) = Cx(t) 


where 
0 0 1 
Del AF a=|9); and C=[0 1]. 


The solution of this system may be written in the form 
t t 
SSC i oP Fe u()49) Bu) dr, 
0 


Since D* = 0, the series definition of the exponential gives 


oP Se u(o)do) = 1 0 
fiulo)do 1 


and therefore 


y(t) = [ff ormcadode 
0 Jt 


t pt 
y(t) = i} i wL(o — t)u(o)u(t)dodt (40.3) 
0 Jo 


or 


1 t>0 
0 t<0° 

Before introducing various types of expansions for the response of nonlinear control systems, let us 
summarize some classical results for the solution of linear differential equations. 


where J is the step function u(t) = | 


40.1.2 Linear Differential Equations 
Let us consider the linear time-varying differential equation 
X(t)=) ai(t)Aix(t), xR", x(0) =xo (40.4) 
i=l 


where for i= 1,...,m, aj: R— R are locally Lebesgue integrable functions and A; are constant n x n 
matrices. We may also write 


m t 
x(t) =x +) > / aj(o)A;x(o) do 
j= 7° 


From the classical Peano—Baker scheme, there exists a series solution of Equation 40.4 of the form [1] 


x(t) =0+D) (f aj(o1) do) A xo + 3 ([ ee aj(o,) a(02)do, do) AiA so 


ij=1 


aa = CR Ee ie i, (01) .. . Oj, (64 )do1 .. i doy) A iy: . Ai,xX0 ++: (40.5) 


solp=l 


This series expansion was used in quantum electrodynamics [11]. 
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Under certain unspecified conditions of convergence, the solution of Equation 40.4 may also be 
written [37] 


x(t) = e2 xq (40.6) 
with 
m 


t m t Ol 
Q(t) = > (/ a(o1)d0) Aj+ ; PS (/ [ a(01)a4(09)doi don) [Ai, Aj] 


i=1 ij=l 


m t fon 02 
+5 by (/ i [ a(1)a(00)4,(09)401dondos ) [Ai, [Aj Ax] 
ijk=1 


1 m t O1 02 
T 3 (/ [ [ (01 )a(02)4(03)do ond) [[AiAj],Axn] +--° 5 (40.7) 
ijpk=1 


where the commutator product or Lie-product is defined by 
[Aj, Aj] = AiAj — Ajj. 
Indeed, for instance, the first terms of the expansion 40.6 are given by 
m 


Laiteane: ye : or be aj(o} )aj(o2)do1do2 
0 2 0 Jo 


i=l ij=l 


2 
1(wo/(f' 
x [Ai Aj] + I (>: (/ a(o1)do1) a) 
Using an integration by parts, this leads to 
m t m t oa 
ye (/ a(o1)do) Ai+ » (/ / a(1)a(00)4a don) AjAj 
0 0 Jo 


i=l ij=l 


which corresponds to the first two terms of the expansion 40.5. 


40.2 Functional Expansions for Nonlinear Control Systems 


The general problem we consider here is how to generalize Equations 40.1 through 40.3, and 40.5 or 40.6 
to nonlinear control systems of the form 


d m 
= = fo(x(t)) + ie uj(t)fi(x(t)) (40.8) 


where fo, fi, ..->fm are C™ vector fields on a n-dimensional manifold M, x takes values in M and u; : Rt > 
R, i=1,...,m are piecewise continuous. In a local coordinate chart, x = (xj,... Xn), Equation 40.8 
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can be written 


M(t) =f(e))+ >) uOfk@M), 1<k<n (40.9) 


i=1 


where the functions f* :R” > Rare C™. Let h €e C*(M). Then 
d ; i 
qe = dh(x)k = Lyh(x(t)) + )° ui(t)Lyh(x(t)), 
i=1 


where Lh is the Lie derivative of h along the vector field fj, i= 0, ...n, 
Lyh= ene) 
‘fi = 1 Ox; : 


Thus, 
m t 
h(x(t)) = h(x(0)) + | uj(o)Lh(x(o)) do 
i=0 


where u(t) = 1, t > 0. Also, 
mat 
Lp h(x(t)) = Lp h(0)) + i) uj(o) Ly Ly h(x(o)) do, 
j=0 
so that 


t 
h(x(t)) = h(x(0)) + so ( [ uj(o) ao) Lg h(x(0)) 


i=0 


‘ [ is uj(o2)uj(o1 Ly Ly h(x(o1))doy doo. 


ij=0 


Iterating this procedure yields 


N m 
h(x(t)) = h(x(0)) + > xl y u(o)do) Lgh(x(0))+ > YO 


V>2 fis. fy=0 


<([ i oye Uj, (Gy)... Ujy (G2) Uj, (1 doy doz .. doy) x Ly, Lp, Ly, h(x(0)) 


+ > A (ee < ae Uiyegx ON1) - + Mj (2)t4j (61) 


+ojN4+1=0 


x a, 7 Ly, Ly, h(x(01))(o1)do,do2...don4y (40.10) 


It is not difficult to see that the remainder [45] 


aan Ve = a Hy x ONL) » > tga) 14 (01) 


+o jN4+1=0 
x ay as Lp Ly, h(x(01))(01)do,do2...don+4 


is such that 
Ants 
Ry || < —————-Cy, 
|Rw || < (VED! 
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where Cy is such that 
fina LG Lh, A(x)| < Cn 
on some compact set and 
A; = sup (1, max i460!) : 


<t<t,l<i<n 


If the vector fields f; are analytic, and the function h is also analytic, then the previous result can be 
strengthened. One can actually prove [12,45] that the series 


m t 
neato) ([ mlodde) Lentstoy) 
i=o WO 


N mt t Oy o2 
+5 ae (ff Sh tC) osu (@n)dor day) Lp, 1p, bt) (40.11) 


V>2 jis -iv=0 
converges to h(x(t)). Indeed in this case, there exists a constant C > 0 such that 
|Lp, «Ly, Lp, h(x)| < (v)IC’. 
forally > 1. 


40.2.1 Volterra Series 


In the following, we consider scalar input, scalar output nonlinear systems on R” called linear-analytic, 


Re =f(x() + ug), x(0) = x0 ine 


y(t) = A(x(t)) 


We always assume that f, g, and A are analytic functions in x, in some neighborhood of the free response 
(when u(t) = 0, Vt > 0). Analyticity is important but the restriction to scalar inputs and outputs can be 
easily removed. We say that a linear—analytic system admits a Volterra series representation if there exist 
locally bounded, piecewise continuous functions 


wn: R" >R, n=0,1,2,..., 


such that for each T > 0 there exists e(T) > 0 with the property that for all piecewise continuous functions 
u(.) with | u(T) |< € on [0, T] we have 


CO at t 
y(t) = wo(t) + pa | te i Wn(t, 01,..-,0n)U(O1)... U(On)do,...don (40.13) 
“2/0 0 


with the series converging absolutely and uniformly on [0, T]. 


40.2.2 Bilinear Systems 


It is not difficult to show that the following class of nonlinear systems called bilinear systems 


Pe =[AW) + u()BO] xO, (0) = x9 aes 


y(t) = c(t)x(t) 


admits a Volterra series representation. 
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Let ®, denote the transition matrix for A(t). Make the change of variable z(t) = ® (0, t)x(t) in order 
to eliminate A. This gives 


fi = u(t)B(t)z(t), 
y(t) = c(t)z(t) 


where B(t)= @,4(0,t)B(t)®,4(t,0) and C(t) =c(t)®,(0,t). Applying the Peano-Baker formula 
(Equation 40.5) construction, we have 


t Z, t 02 ia 7 
= (1+ [ wonBlorrdo + [f° woo2sBCos)u(o1)B(or)4a,don-+---) 210) 
0 0 Jo 
Thus, the Volterra kernels for y(t) are given in triangular form by 


c(t) Pa(t, On)BOOn) Pa (Ons On—1)BOn-1) . . - B(o1) Pa (1, 0)x(0) 
Wnt, 01, 025---5On) : seaeas 
0 if Citp < Op, i,p = 1,2, 3,... 
For A, B, and u bounded this series converges uniformly on any compact interval. 
The existence and the computation of the Volterra series for more general nonlinear systems is less 
straightforward. Several authors [6,15,25,36] gave the main results at about the same time. 
For the existence of the Volterra series, let us recall for instance Brockett’s result: Suppose that 


f(s): Rx R” > R" and g(.,.): Rx R"” > R" 


are continuous with respect to their first argument and analytic with respect to their second. Given any 
interval [0, T] such that the solution of 


x(t) = f(t, x(t), x(0) =0, 
exists on [0, T], there exists an « > 0 and a Volterra series for 
x(t) = f(t, x(t)) + u(t)g(t, x(t), x«(0) =0, (40.15) 


with the Volterra series converging uniformly on [0, T] to the solution of Equation 40.15 provided 
|u(t)| <e. 

Although the computation of the Volterra kernels is given in the previous referenced papers, their 
expressions may also be obtained from the Fliess algebraic framework [13] summarized in the next 
section. 


40.2.3 Fliess Series 


Let us recall some definitions and results from the Fliess algebraic approach [12]. Let u(t), u2(t), ...,Um(t) 
be some piecewise continuous inputs and Z = {z, Z1,...,Zm} bea finite set called the alphabet. We denote 
by Z* the set of words generated by Z. The algebraic approach introduced by Fliess may be sketched as 
follows. Let us consider the letter zy as an operator which codes the integration with respect to time and 
the letter z;, i= 1,...,m, as an operator which codes the integration with respect to time after multiplying 
by the input u;(t). In this way, any word w € Z* gives rise to an iterated integral, denoted by I‘ {w}, which 
can be defined recursively as follows: 


I'{f} =1 
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t 3 
dtl" {v} if w = Zov 
I'{w}= Jo ,veZ*. 40.16 

wl {fi uj(t)dt'{v} ifw=zv ie ( ) 


Using the previous formalism and an iterative scheme like Equation 40.10, the solution y(t) of the 
nonlinear control system 


x(t) =f (x(t) + ye uj(t)gi(x(t)), x(0) = xo 


= (40.17) 
y(t) = h(x(t)) 


may be written [12] 


m 
y(t)=Axo)+ >> So Lg. Lp Ly Wao {zpzj --- Z} (40.18) 
VEO jg jis-+-xiv=0 
with the series converging uniformly for small t and small |u;(t)|, 0 <1 <t, 1 <i<m. This functional 
expansion is called the Fliess fundamental formula or Fliess expansion of the solution. To this expansion 
can be also associated [12] an absoluting converging power series for small t and small |u;(t)|,0 <1 <t, 
1 <i<™m, called the Fliess generating power series or Fliess series denoted by g of the following form 


s=ho) + > DE i lly Meoleuai noe (40.19) 


V=0 jo fls-+.jv=0 


This algebraic setting allows us to generalize to the nonlinear domain the Heaviside calculus for linear 
systems. This will appear clearly in the next section devoted to the effective computation of the Volterra 
series. 

A lot of work uses this formalism, see for instance the work on bilinear realizability [44], some analytic 
aspects and local realizability of generating power series, on realization and input-output relations [49], 
or works establishing links with other algebras [10,17]. 


40.2.4 Links Between Volterra and Fliess Series 


The following result [6,13,31,36] gives the expression of the Volterra kernels of the response of the 
nonlinear control system (40.12) in terms of the vector fields and the output function defining the system, 


oo t pt t 
yi) = wolt) +> f / oy) Wnt, Ta --->T1)U(ty)... u(t )dty- ++ dt, (40.20) 
a1 40 40 0 
where the kernels are analytic functions of the form 


t’ 
wo(t) =D) LA) | = eH A(x), 


v>0 
(t—t)"1)° 
_ vo V1 1 
mieb t= DL EP igh ea) 
Vo.V1 20 
= eT Lge A x9), (40.21) 
Pn. Eph (t—T))™"... uit 
Walts TAs es th) = a gp Lgly --- Lgl, Oa 


Vo.V15"* Vn =O 


Se ge 2 lge thay) 
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In order to show this, let us use the fundamental formula (Equation 40.18). The zero order kernel is 
the free response of the system. Indeed, from Equation 40.18 we have 


t 
wo(t) = h(xo) + Ps > Ly, id Ly, Ly, itso) [ d&j,déj,_ 67s dE jy, 
0 


V=0 jos iv=0 
which can also be written as 
t 
= l a 
y(t) =) Lp ho) 5, 
1>0 
or using a formal notation, 


y(t) = ef h(x). 


This formula is nothing other than the classical formula given in [16]. 
For the computation of the first order kernel, let us consider the terms of Equation 40.18 which contain 
only one contribution of the input u; therefore, 


t t 
i wi(t, t)u(t,) dty = PD LPL eb haa) f dé... d&q dE, d&g... dEq. 
0 0°———~ ——-—$— 


Vo.V1>0 


v,- times vo- times 
But the iterated integral inside can be proven to be equal to 
Ceca kant 
few a) du. 
0 Vv1!vo! 


So, the first order kernel may be written as 


v1 7V0 
(t— 4)" t, 
v1!vo! 


wi(ht)= Y) LLL; h(xo) 


vo,v1 >0 


= etter el Wy h(x), 


For the computation of the second order kernel, let us regroup the terms of Equation 40.18, which 
contain exactly two contributions of the input u; therefore, 


- ee w(t, T1, T2)u(t1)u(t2)dtydt. = > L/°Lg LL gh; h(xo) 


vo.V1,02>0 
xf f° d&o...d&q dE, d&g... dEg dE, d&g... dEq. 
0 /0o—e_—_— ——=—$—S — 
v- times vy - times vo- times 


The iterated integral inside this expression can be proven to be equal to 


é es _ Ww _ V1 iu 
| i (f — t2)” (t2 — 11) 1 u(t 1)u(t2)dt 1 d72. 
o Jo 


va!vz!vo! 


Thus, the second order kernel may be written as 


(t — t2)”(t2 — 4)” 7° 


v2!v1!vo! 


w(ttw= >> Lp? LgL Lgl? h(xo) 
Vo.V1,V2>0 


= et Ege iT ge hl Xp): 


The higher order is obtained in the same way. 
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Using the Campbell-Baker-Hausdorff formula 
oO oi 
oL ol, i 
et Lge “Th(xo) = ) yas Ese 
i=l 


the expressions for the kernels (Equation 40.21) may be written, 
wo(t) = el't h(xo), 


wi(t,t1) = ef Lee (x9) 
st L 
_ i tL 
=, oy adi, Lge't h(x) 
i=1 
W(t, Tas Ta—-1 +9 TL) = etity 7 ear pelt —tILF (xp) ne 


| 


v, 
= = aad Lg ad, Lge" h(xo) 


These kernel expressions lead to techniques which may, for example, be used in singular optimal 
control problems [32]. This will be sketched in the next section. 


40.3 Effective Computation of Volterra Kernels 


Example 40.1 


Let us consider the system [39], 
y(t)+ (w2 + u(t))y(t)=0, t>0, y(0)=0, y(0)= 


After two integrations, we obtain 


yao? [fy odode f° fu o)dodt—t=0 


The associated algebraic equation for the Fliess series (see Section 40.2.3) g is given by 


(l+a 272)9 + 20219 — Zo =0. 


In order to solve this equation, let us use the following iterative scheme 
$=SotSitgs2t---+ gir 
where g; contains all the terms of the solution g having exactly i occurrences in the variable z,, 
go =(1l+o Pgay Z0> 
gi = —(1+ wz)! zozig0 
=—(1+0* ze) leozid +o et 20> 
82 = —(1 +725) "zoz181 


=(1+ wz) ‘zoz (1 + wze)!zoz(1 + wze) 129, 
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Each gj,i > 0 is a (rational) generating power series of functionals y;, i > 0 which represents the ith order 
term of the Volterra series associated with the solution y(t). Let us now compute y;(t) associated with gj, 
i> 0. First, note that 


(1—az) t= > a"eg, aeC, 


n>0 


represents in the algebraic domain, the function e~“’. Indeed, 


Vz) = u 
Ont 


Consider now 
1 1 
= ——_(1+ jwz)7! + ——(1 —jozg)71. 
So ja +jazo) °+ ja jozo) 


Hence, 


1 . . 1 
yo(t) = wo(t) = -——e J + —e™ = — sin(wt). 
2jw 0) 


The power series 


g=—-(+ wz) 1zoz (1 + wz) ‘20, 


after decomposing into partial fractions the term on the right and on the left of z1, 
Liateaye Ma inzo) | | z ae ree eae ae inzo) 
a co) -—~——(1-jw ee co) ——(1-jw 
2jw ee 2jw 10 ‘ 2jo Ios 2jo aie 
or 


gi [(1 + jz) 121 (1 + jozo)! — (1 + jazo)~ 21 (1 — jzo) 7! 


~ foo? 


— (1—jazo)7'2(1 + jozo) | + 1 — jwzo) a —jwz) "| 


In order to obtain the equivalent expression in the time domain, we need the following result [27,31]. 
The rational power series 


(1 — apzo) P°z1 (1 — az) Pz... 21 (1 — ajzo) (40.23) 
where ao, @,..., 4) € C, po, p1,-.-,p) € N, in the symbolic representation of 


t TU T2 
ee af f(t— a). PM — w)fel(tu(t))...u(ty)dy... dt, 


where f(t) denotes the exponential polynomial 
J 
sl) 


For the previous example 


ae ee -1_, 1, ae er =I h<x 
nit) = | =e OE y(t) Fras + al dt— [ —— dott) y(t) Fra + 0 dt. 
9 2jw 2jw 2jw 0 jw 2jw 
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Therefore, 
L 
wie / wieacdade 
0 


with w1(f, t) = =4 sin[w(t — t)] sin oof. 
The higher-order kernels can be computed in the same way after decomposing into partial fraction 
each rational power series. 


40.3.1 Noncommutative Padé-Type Approximants 


Assume that the functions fe :R” — R of Equation 40.9 are C®, with 


PCs). CCD ame CoN 


fls-inZ 0 Do 


Kis k= Aji pi Bio gy. 


ls-+fn20 


Let y denote an equilibrium point of the system (Equation 40.9) and let 


denote the monomial or new state 
(x1) tee (Xn)"", ji aes ‘jn < p- 


Then the Brockett bilinear system 


<p> wan mo ki <p> 

oo ed kat Jk (2 Ui) itseondn Be gee asec ee) , 
Rp ke pe P> 

Yee ree Hin. jn XS pecs jn 


with initial conditions 


(40.24) 


xP. (0) = (yi) ..- Yn) 


Jist Jn 
where 
<p> = 
Naess hse Jn = 
for all j1,...,jn > Oand 
<p> 
Mile 


iff) +...+jn > p, has the same Volterra series up to order p as the Volterra series of the nonlinear system 
(Equation 40.9). 
This system may be interpreted in the algebraic context by defining the generating power series g. a 


associated with a ie and g<?> associated with y<?7, 


<p> _wn ; Hoos ki <p> 

Bish Via dk (Oey Zi ee Caer aes eee) > 
<p> _ : a Pa 

eS ce Bijy,.. sin Sin is vin 


The rational power series g<?> may be seen as a noncommutative Padé-type approximant for the forced 
differential system (Equation 40.9) which generalizes the notion of Padé-type approximant obtained in [4]. 
Using algebraic computing, these approximants may be derived explicitly [38]. These algebraic tools for 
the first time enable one to derive the Volterra kernels. 


(40.25) 
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Other techniques have recently been introduced in order to compute approximate solutions to the 
response of nonlinear control systems. The method in [34] is based on the combinatorial notion of 
L-species. Links between this combinatorial method and the Fliess algebraic setting have been studied 
in [33,35]. Another approach using automata representations [41] is proposed in [23]. 

The Volterra series (Equation 40.13) terminating with the term involving the pth kernel is called a 
Volterra series of length p. In the following we will summarize some properties of an input-output map 
having a finite Volterra series. The main question is how to characterize a state space representation 
(Equation 40.12) that admits a finite Volterra series representation [8]. This study lead in particular to 
the introduction of a large class of approximating systems, having a solvable but not necessarily nilpotent 
Lie algebra [9] (see also [24]). 


40.4 Approximation Abilities of Volterra Series 


40.4.1 Analysis of Responses of Systems 


In this part we show how to compute the response of nonlinear systems to typical inputs. We assume here 
m = 1. This method, based on the use of the formal representation of the Volterra kernels (Equation 40.25), 
is also easily implementable on a computer using formal languages [38]. These algebraic tools allow us to 
derive exponential polynomial expressions depending explicitly on time for the truncated Volterra series 
associated with the response [27,31] and therefore lead to a finer analysis than pure numerical results. 
To continue our use of algebraic tools, let us introduce the Laplace-Borel transform associated with a 


given analytic function input 
t® 
u(t) = ) an al 


n>0 


n 
Su = ) Anz 


n>0 


Its Laplace—Borel transform is 


For example, the Borel transformation of 
cos wt = * pjvt + 5 
is given by 
Su = Ae —jwtzo) 1+ Ae + jotzo)) = (1+ ?22)7}. 
Before seeing the algebraic computation itself in order to compute the first terms of the response to 


typical inputs, let us introduce a new operation on formal power series, the shuffle product. 
Given two formal power series, 


gi= > (gi.w)w and go= >> (go,w)w. 
weZ* weZ* 
The shuffle product of two formal power series g; and gp is given by 
give = >> (gi.wi)(g2,w2)widwe, 
W1,w2EZ* 
where shuffle product of two words is defined as follows, 


e 1l1l=1 
e WrEZ, lOz=wW1 =z 
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e Vz,z' €Z, Vwiw' € Z* 
zwo2'w = z[woz'w'] 4+ z’[zwow’] 


This operation consists of shuffling all the letters of the two words by keeping the order of the letters 
in the two words. For instance, 


2 2 
202102120 = 2292] 20 + 20212021 + 21202120 + 212921. 


It has been shown that the Laplace-Borel transform of Equation 40.23, for a given input u(t) with the 
Laplace-Borel transform g,,, is obtained by substituting from the right, each variable z; by the operator 
Zou]. 


Therefore, in order to apply this result, we need to know how to compute shuffle product of algebraic 
expressions of the form 


Bn = (1 +4020) Zi, (1 +4120) ‘Zin --- (1+ @n—120) Zi, (1+ an20) 


where i, i2,--- ,in € {0,1}. This computation is very simple as it amounts to adding some singularities. 
For instance 


(1 +.az)~!O(1 + bz)! = (1 + (a+ b)zo) I. 


Consider two generating power series of the form (Section 40.4.1), 


Sp =(+ agzo) 1zj,(1 +4120) *z;,.-.+ ap—120) ‘zi, (1 + apz) | 


and 
Bq = (1+ Boz) "Zz, (1 + biz) Zp «(+ bg— 120)” zg (1 + bg20)* 


where p and q EN, the indices i}, i2,..., ip € {0, 1}, ji, ja, --->fq € {0, 1} and aj, bj € C. The shuffle product 
of these expressions is given by induction on the length 


Bp OBq = pOBq—1Zq(1 + (ap + bq)zZ0) | + Sp—10SqZip (1 + (ap + bg)Z0) 


See [30] for case-study examples and some other rules for computing directly the stationary response 
to harmonic inputs or the response of a Dirac function and see [14] for the algebraic computation of the 
response to white noise inputs. This previous effective computation of the rational power series g and of 
the response to typical entries has been applied to the analysis of nonlinear electronics circuits [2] and to 
the study of laser semi-conductors [18]. 


40.4.2 Optimality 


Volterra series expansions have been used in order to study control variations for the output of nonlinear 
systems combined with some multiple integral identities [10]. This analysis [32,42,43], demonstrates links 
between the classical Hamiltonian formalism and the Lie algebra associated with the nonlinear control 
problem. To be more precise, let us consider the control system 


x(t) =f (x(t), u(t), 
xe fe ae: (40.26) 


where f : R” x R” — R" is a smooth mapping. Let h: R” — R be a smooth function, let 


y(t, xo) = ef xo 
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be the free response of the system and let x(t, xo, u) be the solution relative to the control u, u being 
an integrable function taking values in some given bounded open set U € R™. An example of control 
problem is the following: find necessary conditions such that 


h(y(T)) = min h(x(T, xo, u)). (40.27) 
Let wo : [0, T] x R” — R be defined as follows, 
wo(t, x) = h(eF OF x), 
It is easy to see that the map 2 : [0, T] > (R)* given by 
Ow 
nt) = (6 vo) 
Ox 


is the solution of the adjoint equation 


A) = X(t) ote 0) 


(t, y(t) 


and X(T) = dh(y(t)). 
A first order necessary condition is provided by the application of the Maximum Principle: If y(f) 
satisfies (Equation 40.27) for t € [0, T], then 


ad,, fiwolt, y(t))=0 fort € [0, T] (40.28) 
and the matrix 


((fgwo(t, v(t)))) (40.29) 


is a non-negative matrix for t € [0, T] where 


= pel 
fox) =fOx), fix) = 5-6) 


and 


2. 
Sfig(x) = it pita i,j=1,...,m. 


The reference trajectory y is said extremal if it satisfies Equation 40.28 and is said singular if it is extremal 
and if all the terms in the matrix (Equation 40.29) vanish. If y is singular and it satisfies Equation 40.27, then 
it can be shown for instance (see [32]) that if there exists s > 1 such that for t € [0, T] andi,j=1,...,m, 


[ads \fi,adf filwo(ts y(t) =0 fork =0,1,s—1, 
then 
[ad fade filwo(t, y(t) =0 for k,k: = 0 with ky +k, =0,...,2s 
and the matrix 
(lads fi, adf flwo(t, v(0))) ) 


is a symmetric non-negative matrix for t € [0, T]. 
Asa dual problem, sufficient conditions for local controllability have been derived using Volterra series 
expansions (see for instance [3]). 
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40.4.3 Search for Limit Cycles and Bifurcation Analysis 


The Hopf bifurcation theorem deals with the appearance and growth of a limit cycle as a parameter is 
varied in a nonlinear system. Several authors have given a rigorous proof using various mathematical tools, 
series expansions, central manifold theorem, harmonic balance, Floquet theory, or Lyapunov methods. 
Using Volterra series [47] did provide a conceptual simplification of the Hopf proof. In many problems, 
the calculations involved are simplified leading to practical advantages as well as theoretical ones. 


40.5 Other Approximations: Application to Motion Planning 


Let us consider a control system 


k= So ui(t)fi(x). (40.30) 


i=1 


The dynamical exact motion planning problem is the following: given two state vectors p and q, find an 
input function u(t) = (u1(t), u2(t), ..., Um(t)) that drives exactly the state vector from p to q. In order to 
solve this problem, several expansions for the solutions which are intimately linked with the Fliess series 
are used. 

When the vector fields fj, i = 1,..., mare real analytic and complete and such that the generated control 
Lie algebra is everywhere of full rank and nilpotent, then in [26,46], the authors described a complete 
solution of the previous control problem using P. Hall basis. 

Let A(z1,Z2,...;Zm) denote the algebra of noncommutative polynomials in (z1,Z2,...,Zm) and let 
L(Z1,Z2;.--,Zm) denote the Lie subalgebra of A(z1,2Z2,..., Zm) generated by z1,Z2,...,Zm with the Lie 
bracket defined by [P, Q] = PQ— QP. The elements of L(z1,22,...,Zm) are known as Lie polynomials 
in Z1,2Z2,...,Zm. Let Fm be the set of formal Lie monomials in z1,Z2,...,Zm. A P. Hall basis of L(z1, z2, 

.->2m) is a totally ordered subset (B, <) of Fin such that, 


The z; belong to B. 

If A, B € B, and degree(A) < degree(B), then A < B. 

If P € F,, and P is not one of the z;, then P € B if and only if P = [A, B] with A, B € B, A < B, and either 
B= z; for some i or B=[C, D] with C, De B,C <A. 


If Lg(Z1, Z2,.--,Zm) denote the nilpotent version of L(Z), Z2, .. ., Zm) obtained by killing all the monomials 
of degree k + 1, it is not difficult to see that {M € B: degree(M) < k} is a basis of Ly (z1, Z2,..-;Zm), where 
B is the set of all elements of L(z1, Z,...,Zm) obtained by actually evaluating the Lie brackets. 

Now, let 21,..-5 Zn» Zm+i>--->Z, a P. Hall basis of Ly(z1, Z2,...,Zm) and let Ef be the evaluation map 
that assigns to each P the vector field obtained by plugging in the f;,i=1,...,m, for the corresponding 
z;. We assume that the vector fields fin41,..., fr are given by fj = E,(zj) forj=m+1,...r. 

An expansion for Equation 40.30, and consequently a solution to the exact motion planning problem, 
is then obtained [26] from the solution of 


S(t) = S(t) (uz + w2(t)za+---+u-(Hz), S(0)=1, 


written as a product 


S(t) = elir(t)zr pltra@zr-1 elia(t)z2 ol (Oz | 
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For example, in the case k= 3, m=2, we may choose z3 = [Z1, 22], Z4 =[21,[Z1,22]] and z5 = 
[Z2, [Z1, Z2]]. For this choice, the functions hj, j=1,...,r are computed by solving 


hy =u, 
ho =u 
hg =hw.+ 13 


. 1 
hg = shine +hyu3+uq4 


hs =hou3 +hyhou 


with hj(0) =0,j=1,...,5. 

In order to take into account systems with drift, for the system (Equation 40.30) where u(t) may be 
identically equal to 1, a different basis, called the Lyndon basis, has been used in [21]. Without going 
into the details, for the previous case, it is shown that the solution is given by the following exponential 
product expansion 


x(t) = esi fi 68200 f2 p83 (Of pS4(Of4 p85 (Of5 Id(x)|x() 


with here 23 = [22,21], 24 = [22, [z1, Z1]], 25 = [22, [22,21], fj =Ep(@) for j = 3,...,5 and 


t 
n= | melds 
0 
t 
n= | ads 
0 
t 
t= | E> dé, 
0 


t 
u= | §3 dé) 
0 


1 t 
t= feat. 
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41.1 Introduction 


Integral quadratic constraints (IQC*) are inequalities used to describe (partially) possible signal com- 
binations within a given dynamical system. IQC offer a framework for abstracting “challenging” (e.g., 
nonlinear, time-varying, uncertain, or distributed) elements of dynamical system models to aid in rigor- 
ous analysis of robust stability and performance (more specifically, to establish Lz gain bounds, passivity, 
and other system properties, which can be expressed, exactly or approximately, in terms of general- 
ized dissipativity). While the technique can be employed to prove general theorems, it is most powerful 
when used to derive optimization-based algorithms for certification of stability and robustness of specific 
feedback systems. 

IQC can be viewed as implicit generalized dissipation inequalities with known quadratic supply rates 
and unspecified storage functions, not necessarily quadratic or sign definite. Alternatively, they have a 
frequency domain interpretation as bounds on the degree of harmonic distortion produced by a specific 
element of the complete model. The past research in nonlinear systems and robust control can be harvested 
to extract rich IQC descriptions of commonly used components of feedback systems. These IQC can then 
be reused in a modular approach to system analysis. 

The IQC framework is closely related to multiplier-based passivity, upper bounding of structured sin- 
gular values, quadratic relaxations in nonconvex optimization (including the sums of squares approach 
to positivity of multivariable polynomials), and other constructive techniques for handling nonlinearity 
and uncertainty. The IQC approach could be viewed as a unifying framework where ideas from sev- 
eral research directions have been combined and developed in order to obtain a flexible and powerful 


* 


We will use “IQC” for both singular and plural forms. 
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* Multipliers 


* Performance 


* Software 


1QC-theory 
Influenced by all fields 
a unifying approach 


Dissipative systems theory Convex optimization 


* Storage function * Semidefinite programming 


Absolute stability theory * Linear matrix inequality 
* Supply rate 


* Quadratic forms * Sum of square 


* Dissipation inequality : , 
* S-procedure 


* KYP lemma 


FIGURE 41.1 The IQC theory unifies powerful ideas from several important research fields: (1) The input-output 
theory; (2) the dissipative systems theory that was developed primarily in the west during 1960-1975; (3) The absolute 
stability theory that was developed in the Soviet Union during 1960-1975; (4) The robust control field from 1980-1995, 
and finally; and (5) tools from convex optimization that have been developed since the late 1980s. 


framework that can be easily implemented as a MATLAB®-based software package. This is illustrated 
in Figure 41.1. 


41.1.1 Notation 


The real numbers are denoted by R and the complex numbers by C. The subset of nonnegative real 
numbers is denoted R;. We treat elements of R?@ (or C4) as d-by-1 column matrices. The notation [x; y] 
means the column vector [x y! ik obtained by stacking two column vectors x and y on top of each other, 
and’ is the operation of Hermitian conjugation (transposition for real matrices). 

Central to the development in this article is the use of quadratic forms. 
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41.1.1.1 Quadratic Forms 


A quadratic form o : R4++ Risa function defined by o(x) = x'Px where P is a real d-by-d matrix (the 
set of all such matrices to be denoted by R4*“). Without loss of generality, P can be assumed to be 
symmetric, that is, such that P= P’. The Hermitian extension o% : C4++ R of o is then the function 
defined by o(x) = x’ Px for all x € C4. Any function that has the same form as that of o% is called a 
Hermitian form. 

The notation o > 0 means that the quadratic form is positive semidefinite, that is, o(x) > 0 for all 
x € R@, This is equivalent to P being positive semidefinite, which is denoted as P > 0. 


41.2 Getting Started with IQC 


In this section we introduce basic elements of the IQC framework by applying it to the classical problem of 
global analysis of a feedback interconnection of a single-input single-output (SISO) linear time-invariant 
system (LTI) and a memoryless “rate limited” nonlinearity. The overall model can be described either by 
the block diagram 


—+ G(s) 


o(-) *#— 


or by the n-dimensional ordinary differential equation 
x=Ax+Bw, w=(Cx), (41.1) 


where A, B, C are given real matrices with A Hurwitz (i.e., det(s! — A) 4 0 for Re(s) > 0), G(s) = C(sI — 
A)~'B is the transfer function of the LTI subsystem, and @: Rr R is such that (0) = 0 and b € [0,1]. 
The analysis objective is to establish global asymptotic stability of the equilibrium x = 0. 


41.2.1 IQC Modeling 


IQC analysis begins with IQC modeling, which includes (1) recognizing the exact model S of the dynamical 
system under consideration; (2) specifying the analysis objective as an IQC to be established for S; and 
(3) finding a sufficiently rich family of IQC readily known to be satisfied on S. 

To apply IQC analysis to the system in Equation 41.1, let S be its behavioral model in terms of signals 
w and x: 


S= {[w;x]: x= Ax+ Bw, w= o(Cx)}. (41.2) 


41.2.1.1 Complete and Conditional IQC 


For a set S = {q} of d-dimensional signals q = q(t), an IQC is defined by a quadratic form* o: R4++ R 
(referred to as the supply rate of the IQC), and a statement of the existence of a lower bound k = k(q(0)) 


* While it is possible to construct a similar theory utilizing general supply rates o, requiring o to be quadratic is important 


for practical feasibility of the approach: eventually, checking positive semidefiniteness of linear combinations of different 
o will be required to reach an analysis conclusion, and quadratic functions form the only generic class for which this can 
be done efficiently. 
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for the integrals of o(q(t)) over long intervals of time: a complete IQC o > 0 on S means existence of a 
continuous function Kk: R? +> R, such that k(0) = 0 and 


T 
o(q(t)) dt = —K(q(0)) (41.3) 
0 


for all gq € S and T > 0, while a conditional IQC o > 0 on S states that 


} o(q(t)) dt > —K(q(0)) (41.4) 


for all signals q € S of finite energy. 

Informally, one can think of a complete IQC o > 0 as an implicit dissipation inequality o(q(t)) = 
dV (x;(t))/dt where x; = x;(t) is the hidden state of the system, and V = V(x;) is an unknown non- 
negative storage function satisfying an unknown* upper bound V(x,(0)) < k(q(0)). In a similar way, a 
conditional IQC o > 0 corresponds to the case where the storage function is not necessarily nonnegative, 
but satisfies V(0) > 0. 


41.2.1.2 IQC as Analysis Objective and Background Information 


IQC can be used to define an objective of system analysis. For example, since A is a Hurwitz matrix, global 
asymptotic stability of the equilibrium x(0) = 0 in Equation 41.1 is implied by the energy bound 


T 
i: Iw(t)2dt <K(x(0)), YT >0, (415) 
0 


where k : R” ++ R4 is continuous and such that «(0) = 0. By definition, Equation 41.5 is a complete IQC 
0, > 0 on S, where 
o,([w;x]) =—|wl? (w eR, x ER"). (41.6) 
In the standard IQC analysis flow, the main effort in proving 0, > 0 is devoted to finding a quadratic 
form oo such that o9 > 0 and ox > 09. This is accomplished by compiling a large set Ag = {0} of quadratic 
forms for which the IQC o > Oare either trivial or readily established, and then optimizing over the convex 
hull A of Ag to satisfy o, > o for some o € A. Thus, the IQC ob 0 foro € Ag become elementary pieces 
of “background information” about S, and success of IQC analysis depends on one’s ability to generate a 
set Ag which is rich enough. 


41.2.1.3 Sector IQC 


Since (0) = 0 and b € [0, 1], p(v) is between 0 and v for all v € R, that is, the points [v; w] € IR? such that 
w = 0(¥) lies in the sector w(v — w) > 0. 


w(t) 


— > o() TT " * 


Hence, w(t)[ v(t) — w(t)] > 0 for all t and all [w;x] € S, v = Cx, which implies that the complete IQC 
0; > O is satisfied on S for 


o,([w;x]) =2w(Cx—w) (weR, x € R”). (41.7) 


The corresponding lower bound k = k; can be chosen as k; = 0. 


* In most applications, it is possible to have explicit upper bounds «(-), which can be useful in performing advanced analysis 


tasks. 
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41.2.1.4 Popov IQC 


The sector IQC o; > 0 does not reflect the time-invariant nature of the relation between v = Cx and 
w: it is still satisfied when the equation w(t) = $(v(f)) in the definition of S is replaced by w(t) = 
a(t)! (a(t)v(t)), where a = a(t) is an arbitrary time-varying nonzero coefficient. Two less obvious IQC 
can be derived from the observation that for every [w; x] € S and v = Cx we have 


h 
w= Bria. where W(h) = b(t) dt. 
dt 0 


Since, due to (0) =0 and b € [0,1], we have 0 < W(v) < v*/2 for all v ER, the IQC Op > 0 (with 
K([w;x]) = |Cx|? for w € R, x € R”) and —6p > 0 (with k = 0) hold on S for 


op([w; x]) = 2wC(Ax + Bw) (weR, xe R"). (41.8) 


41.2.1.5 Pure Integrator IQC 


Since, due to Equation 41.1, Ax + Bw is the derivative of x, for every symmetric real matrix Q = Q’ the 
signal 2x’Q(Ax + Bw) is the derivative of x/Qx. Hence, both o; > 0 on S for all Q= Q ando; > OonS 
for all Q = Q = 0, where 


o7([w;x]) = 2x’Q(Ax+ Bw) (weER, xe R”), (41.9) 


with k([w; x]) = x’Qx. 


41.2.1.6 Zames-—Falb IQC 


To give a taste of less obvious IQC relations describing the memoryless nonlinearity of system in Equa- 
tion 41.1, consider the following IQC from a (much larger) family established by a classical result by 
G. Zames and P. Falb (see Section 41.4.2 for details). Define an extension S, of S by 


Se = {[wixs§s8]: [wsx] € S, £+3§ +2 = 2w, £(0) = (0) = 0}. 
It turns out that, due to the rate limit b € [0, 1], the complete IQC 0, > 0 is satisfied on S,, where 
Oz(Lw; x; 0; 1]) = 2(Cx —w)(w— Eo) (w, €0,&1 €R,x € R"), (41.10) 


with k = 0. 

This IQC has an interesting interpretation in the frequency domain. Assume that v = Cx has finite 
energy, and let y= v—w =v — (v). It can then be shown that the IQC 0, > 0 claims that, due to the 
rate limit imposed on the memoryless nonlinearity >, the integral of Re[w(jw)’Ge(jw)~(jw)], where 
Ge(s) = (s* + 3s)/(s* + 3s+2) and w, y denote the Fourier transforms, is also not negative. In other 
words, the IQC 6, > 0 sets a limit on the amount of harmonic distortion, which can be introduced by 
the nonlinear transformation v +> (v). Note that this frequency domain interpretation is far from trivial 
since in general, the time domain relation w(t)y(t) > 0 does not imply the corresponding inequality 
Rel w(jw)'y(jo)] = 0 for the Fourier transforms. 

While it can be shown that the IQC o, > 0 indicates existence of a nonnegative storage function 
V, = V.(E,&) such that (Cx — w)(w — 2) > dV, /dt, no explicit form for V; is available.* Despite its 
“implicit” nature, the IQC 0, > 0, can be very useful in analysis of specific nonlinear systems. 


* To the best of authors knowledge, the only available proof is based on a converse storage function argument, and, as such, 


is not very constructive. 
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Note that applying the “extension” S +> S, does not remove any of the original IQC: the sector and 
Popov IQCo; > 0,6 > 0, —op > Oare still valid on Se*, the “objective” IQC o, > 0 has the same meaning 
on S and &,, and the family of “pure integration” IQC o; > 0 generalizes to oj. > 0, where 


/ 


x Ax + Bw 
Ofe(Lw3 x3 §031]) =2] G0 | Q 51 =O (41.11) 
51 —2& — 3; + 2w 


41.2.1.7 Combining IQC 


As long as linear operations are concerned, IQC can be handled as usual inequalities: if o, > 0 and 02 > 0 
on S then cjo; + c202 > 0 on S for arbitrary nonnegative real numbers c;. Similarly, if o; > 0 and o2 > 0 
then co; + c202 > 0. This allows the user of the IQC framework to convert individual IQC satisfied for 
subsystems of a larger model into one convex set A = {o} of quadratic forms o defining valid conditional 
IQC of 0 on S. In addition, since many of the classical IQC are readily shown to be complete, a convex 
subset A+ C A such that o > 0 for allo € A+ can be constructed as well. 

In particular, for the behavioral model S defined by Equation 41.1, the set A compiled so far is given by 


A = {cs05+ Cpop toy: cs E Ry, cp ER, Q=Q}, (41.12) 


where 65, 6p, 0; are defined in Equations 41.7 through 41.9. In contrast, the set A of recognized complete 
IQC consists of o = c,05 + CpSp + 67 with c; > 0, cp > 0, Q= Q’ > 0. It is important to understand that, 
depending on the coefficient matrices A, B, C, there could be some o € A such thato > 0onS buto ¢ A+: 
those are the quadratic forms defining valid complete IQC which are not recognized as such. 


Similarly, for the extended model S,, 
A = {¢505 + Cp0p + Of + Cz0z: Cs,Cz ER, Gp ER, Q = Qi}, (41.13) 


with o7, defined in Equation 41.11, and the forms from A satisfy the additional constraints cp > 0 and 
Qe = 0. 


41.2.2 Feasibility Optimization and Postfeasibility Analysis 


Once the exact model S is defined, the objective IQC o, is selected, and a convex set A = {o} of valid 
IQC o> 0 on S is compiled, the next step in the IQC framework is feasibility analysis, understood as 
the search for oo € A satisfying 0, > 09. In a typical application, there is also a “cost” parameter to be 
minimized (in the IQC model for Equation 41.1, this could be y, subject to yI = |cp|C’C + Q, bounding 
k in Equation 41.5), so there is a well-defined optimization criterion in addition to the feasibility task. 

If there is no o9 € A such that 0, > 09, the IQC analysis fails, and the only option is to return to the IQC 
modeling step to find a larger set A. Sometimes it helps to consider (as in the Zames—Falb IQC example) 
replacing S with an extension S, of S: a cleverly defined set S, = {qe} of signals of fixed dimension d, > d 
such that for every q € S there is at least one signal g such that [q; g] € Se. 

Having 0, > 09 for some 09 € A means that o, > 0, that is, either o, > 0, as desired, or 0, > 0 but 
o, % 0. In most applications, the latter means that the system is strongly unstable: subject to mild assump- 
tions, the conditional IQC o, > 0 on S is a “certificate of dichotomy,” that is, it implies that S is either 
quite stable or very unstable, no middle ground. In most situations, general theorems of postfeasibility 
analysis provide easy criteria to decide which outcome actually takes place. 


* Here we allow some abuse of notation, using the same identifier for a quadratic form o: R41 R and its “extension” 


&: R4+2 & R defined by o([w; x; 93 &1]) = o([ 3 x]). 
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41.2.2.1 Feasibility Optimization and Semidefinite Programming 


When the set A is linearly parameterized by a vector variable which is in turn subject to a linear matrix 
inequality (LMI) constraint, the search for 09 € A satisfying 0, > 69 becomes a semidefinite program. 

For example, in the analysis of Equation 41.1 with A defined by Equation 41.12, one possible semi- 
definite program to solve is: min y subject to yI = +cpC’C + Q, cs = 0, and 


| 2e;= 1 = 2¢CB Bey aan (41.14) 


—csC’ — cpA'C’ — QB —QA—A’Q 
where the decision variables are y, cp, cs € Rand Q= Q’ € R”*", While the LMI in Equation 41.14 looks 


quite cumbersome, the quadratic form notation is usually much more straightforward. For example, the 
functional version of Equation 41.14 is 


- ||? — 2csw(Cx — w) — 2cpwC(Ax + Bw) — 2x'Q(Ax + Bw) > 0V w,x. 


Certain programming tools can be used to convert quadratic form notation into the format of the standard 
semidefinite program solvers [6]. 


41.2.2.2 Feasibility Optimization and the Kalman-Yakubovich-Popov Lemma 


While conversion to a semidefinite program, to be solved numerically, appears to be the only practical 
approach to feasibility analysis of advanced IQC models, valuable theoretical insight can be gained by 
applying the following version of the classical Kalman-Yakubovich-Popov (KYP) Lemma (sometimes also 
called the positive real Lemma). 


Theorem 41.1: 


For real matrices A, B of dimensions n-by-n and n-by-m respectively, and a Hermitian formo: C™'" +> R 
with real coefficients, let Q be the set of wo € R such that the matrix Aw = jo, — A is invertible. For wm € Q 
let Ly = A;,'B. Consider the following statements: 


a. o([u;Lyu]) > 0 for allue C™ and w € Q 

b. There exists w € Q such that o([u; Lu) > 0 for allue C", u #0 

c. The quadratic form o([u; x]) — 2x'Q(Ax + Bu) (where x € R" and u € R") is positive semidefinite 
for some real matrix Q = Q' 


Then 


i. (c) implies (a) 
ii. when the pair (A, B) is controllable, (a) implies (c) 
iii. when the pair (A, B) is stabilizable, (a) and (b) together imply (c) 


Theorem 41.1 can be used to formulate frequency domain conditions of feasibility in IQC analysis 
when the “pure integration” term 2x’Q(Ax + Bu) is present in the description of A. 
41.2.2.3 The Circle Criterion 


In the special case where only the pure integrator and sector IQC are used to represent system Equa- 
tion 41.1, the feasibility analysis calls for finding c,; € Ry. and Q = Q’ such that 


—|w|? — 2c.w(Cx — w) — 2x’Q(Ax + Bw) >0 VweR, xeER". (41.15) 


Theorem 41.1 offers valuable insight into the feasibility of the associate semidefinite program: together 
with the identity CL,, = G(jw), it allows one to conclude that Q = Q’ satisfying Equation 41.15 exists for 
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some cs € R, if and only if (assuming A is Hurwitz and (A, B) is controllable) 
—|w|* —c.Re[G(jw) — lw? >0, YweC,weR 


Since c, can be taken arbitrarily large, it can be seen that the frequency domain condition holds if p < 1, 
where p = sup, cp Re[G(jw)] (the minimal upper bound of Re[G(ja)] over w € R). 

When p > 1, the IQC analysis proves nothing: system in Equation 41.1 may be stable or unstable 
depending on fine details not reflected in the IQC model used so far. When p < 1, we have o, > og where 
69 =c,0, +0, for some c, > 0 and Q= Q’. Since 6, > 0 on S and o; > 0 on S for Q > 0, the desired 
complete IQC o,. > 0 will be established as long as it is possible to guarantee that Q = Q’ is positive 
semidefinite. When Q is found numerically by solving a semidefinite program, the inequality Q > 0 can 
be checked directly. Otherwise, checking that Q > 0 becomes a typical postfeasibility analysis task. 

Substituting w = 0 into the inequality in Equation 41.15 yields QA + A’Q < 0. Since A is a Hurwitz 
matrix, which in turn implies Q > 0, recovering the CIRCLE CRITERION: system in Equation 41.1 is globally 
asymptotically stable when Re[G(jw)] < 1 for all w € RU {oo}. 


41.2.2.4 The Popov Criterion 


In the case where the pure integrator, sector, and Popov IQC are used together to represent system in 
Equation 41.1, the feasibility analysis calls for finding c, € Ry, cp € R, and Q = Q’ such that 


—|wl|? — 2c,w(Cx — w) — 2cpwC(Ax + Bw) — 2x'Q(Ax + Bw)>0 VweR,x eR". (41.16) 


Theorem 41.1 can be used to show that existence of c; €R+, cp €R, Q= Q' €R"*” satisfying in 
Equation 41.16 is equivalent to existence of p € R such that pp < 1, where pp is the minimal upper bound 
of Re[(1 + jap)G(jw)] over w € R (when pp < 1, one can use cp = pc; with c; > 0 large enough). 

Since the storage function for the IQC 6; + pop + 0; > 0 is V(x) = x’Qx + 2p(Cx), the desired IQC 
0, > 0 is established when nonnegativity of V is assured. Since Equation 41.16 with w =0 implies 
QA + A’Q < 0, we know that Q > 0 is positive semidefinite. Since (y) > 0 for all y € R, this implies 
V > 0 when p > 0. 

The postfeasibility analysis becomes trickier in the case p < 0. The upper bound y(y) < y?/2 can be 
used to show that V > 0 whenever Q+ CpC" C > 0. However, this special treatment is not necessary, 
as, according to the general postfeasibility analysis theorems discussed in Section 41.3.2, the inequality 
Ox = Cs0s + CpSp + 7 implies the complete IQC cso5 + cpop + 67 > 0 whenever c; > 0 and A is a Hurwitz 
matrix. This establishes the classical POPOV CRITERION: system in Equation 41.1 is globally asymptotically 
stable when there exists p € R such that Re[(1+ pjw)G(jw)] < 1 for all w € RU {oo}. 


41.3 Theory of IQC Analysis 


In general, IQC analysis follows the steps highlighted in Section 41.2 (IQC modeling, feasibility optimiza- 
tion, postfeasibility analysis). This section gives a formal presentation of the framework. 


41.3.1 IQC Modeling 
An IQC model consists of a system (understood as a set S of signals of fixed dimension d) and an analysis 


objective in terms of IQC, where each IQC is a property of S defined by a quadratic form o on R4. 


41.3.1.1 Signals 


A continuous time (CT) signal of dimension d is an element of the set £4 of all measurable locally bounded 
functions q: [0,00) IR¢ (we also use L for £4 with d = 1). 
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We define q = dq/dt for q € C4 (in which case the constraint dq/dt € £4 is imposed automatically). 
We use Z[q] to denote the total integral of q: 


lo) 
ria\= [ q(t) dt for qe £4 
0 


(Z[q] is not defined for some q, and it is possible to have Z[q] = oo or Z[q] = —00). We use ||q|| to denote 
the square root of the energy ||q\|? = Z[|q|7] of signal g, and denote by i the set of all finite energy signals 
in £4. Given T > Oandq€ L4,p=[qlr € ro denotes the past history of q: p(t) = q(t) for t < T, p(t) =0 
fort > T. 

When: R4+> Risa quadratic form, we use o(q) as a shortcut notation for Z[o(q)] when q € £4. In 
the special case o(x) = |x|* (x € R?@), we use IIxll% in place of o([x]r) for x € L4 and T > 0. 


41.3.1.2 Systems 


Mathematically, we view a general system as a behavioral model, that is, simply a set S of signals of 
given dimension. The elements of S are assumed to represent all possible combinations of all signals of 
interest (the “outputs” may include components of the hidden state as well as auxiliary signals introduced 
specifically to simplify the analysis). Accordingly, a system is a subset of £7, where d is a positive integer. 

We also consider the special case of operator models, which are functions A: L*‘ +> £” mapping 
signals to signals. An operator model A can be described by a behavioral model defined as the graph 
Ga = {[A(v);v]: ve Ly of A. The operator model A is called causal when [A(v)]r is completely 
defined by [v]r for all T > 0, that is, [A(v1)]r = [A(v2)] 7 whenever [v,] 7 = [v2] r. The operator model 
A is called stable (in the “bounded input energy implies bounded output energy” sense) if || A(v)|| < 00 
whenever ||v|| < oo. 

In particular, a finite order LTI operator defined by real matrices A, B, C, D is a causal operator model 
mapping v to y= Cx + Dv € L where x is defined by « = Ax + Bv and x(0) = 0. An LTI operator is 
completely defined by its transfer matrix G(s) = D + C(sI — A)~!B, an element of the set RL‘*” of all 
real rational k-by-m matrix functions of scalar complex argument s which are proper (i.e., bounded as 
|s] > 00). 


41.3.1.3 Definition of IQC 


Leto: R47+> R bea quadratic form. We say that system S C £4 satisfies the conditional IQC defined by 
o (shortcut notation o > 0 or o(q) > 0 when it is more convenient) if there exists a continuous function 
kK: RIB R, such that k(0) = 0 and 


o(q) > —K(q(0)) whenevergeS,  ||q|| < 00. (41.17) 


We say that system S satisfies the complete IQC defined by o (shortcut notation o > 0 or o(q) > 0) if there 
exists a continuous function k: R41 R+, such that «(0) = 0 and 


o([q]r) => —k(q(0)) wheneverqgeS, T>0. (41.18) 


41.3.1.4 Extended Systems and IQC 


Introducing new signals into consideration (an action referred to as extension here) frequently helps 
to expose IQC relations in the original model. In general, the behavioral model S c L4*N is called an 
extension of Sp C £4 if for every go € So there exists g € LN such that [qosglES. 


Example 41.1: 


Consider the behavioral model So = {[u?;u2]: u € L} (it is the graph of the memoryless nonlinear 


operator v +> v2/3), It can be shown that every quadratic form o: R2 + R such that o 0 on So is 


41-10 Control System Advanced Methods 


positive semidefinite, that is, Sg does not satisfy any nontrivial IOC. The extension S = {{u3; usu; 1]: 
u € £L}, however, satisfies a set of useful IQC o > 0, where 


o([X3;X2;X1;X0]) = C1 (xox2 —x?) + €2(Xox3 — X1X2) +.€3(x1x3 — x3), 


and the coefficients cj € R are arbitrary. 


41.3.1.5 Stable LTI Extensions and Frequency Weighted IQC 


The situation in which an extension S of So is defined by a stable LTI operator E according to S= 
{Lqo; Ego] : qo € So} is of special importance in the IQC framework. Allowing some abuse of notation, 
let E = E(s) also denote the transfer matrix of E. Assume that o> 0 on S for some quadratic form 
o: RAY GR. 

Let II = P{o, E} € RL4*¢ be the transfer matrix defined (uniquely) by the identities 


o4 ([u; E(jo)u]) =u M(jo)u, T(jo)=M(jo)’ Vue C4, weR, 


where o/ denotes the Hermitian extension of o. The IQC o(q) > 0 on the extended system S can be inter- 
preted as a frequency domain weighted IQC on the imaginary axis (shortcut notation of ([qo; E(jw)qo]) > 0 
or qyI1(jw)go > 0). Indeed, due to the Parseval identity, o(q) can be represented as the integral 


CO 
J Gnlioy jw q0(j0) do 
—0o 
for all qo € £4, q = (qo; Eqo], where qo denotes the Fourier transform of qo € £4. While TT is uniquely 
defined by E and o, a single I corresponds to a variety of pairs (E, 0). All such “realizations” are equivalent 
in the framework of conditional IQC. 


41.3.1.6 Extended IQC for I/O System and State-Space Interpretation 


In the most common scenario, extended IQC are derived for input/output relations, that is, for behavioral 
models of the form 
So = {qo =[wsv]: ve Ck, wel”, w=A(v)}, 


where A: L* ++ £ isa causal and stable operator. Suppose the transfer matrix E in Section 41.3.1.5 has 
a state-space realization E(s) = C,(sI — A.) ! [Be Bea| + [Dei Dea]; where Ag, Bei, Ber, Ce, Det, Der 
are fixed real matrices, and A, is a Hurwitz N-by-N matrix. Then the stable dynamical extension S can 
alternatively be represented as 


Ss = (q= [wi vixe] © LVN | w= Aly), ke = Acxe + Bary +Beaw, Xe(0)=0}, (41.19) 
(see Figure 41.2 for an illustration) and the corresponding quadratic form is 
O;5(q) = o([w; V; CeXe + Dev + Deow])) 


Clearly o,; > 0 on S,; ifand only if o? ([qo; E(jw)qol) >0on So. 

For example, in the special case of the Zames—Falb IQC in Section 41.2.1.6, the IQC 0, > 0 on S& 
can be expressed as a frequency-weighted IQC (v — w)(w — H(jw)w) > 0, where H(s) = 2/(s? +3s+2), 
relating signals v = Cx € Lo and w = o(y) € Lo. In this case, the explicit expression for I is 

esis 0 1— H(jow) 
a , —H(jw)' 2Re[1— on 
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we, =Apx,+ Bev + Boe Ve 


Ve= Coxe iP Dav +Dow 


FIGURE 41.2 Extended IQC setup for I/O system A. 


41.3.2 Feasibility Optimization 


This section briefly discusses the problem of feasibility optimization. The attention is restricted to a special 
system structure that frequently appears in applications. 


41.3.2.1 Stable Operator Feedback Setup 


An important class of IQC analysis scenarios is given by the stable operator feedback setup which calls 
for certifying the complete IQC r|f|? — |e|? > 0 (with r > 0 being as small as possible) in the feedback 
interconnection of a stable causal operator A (input v € L*, output w € £”) and a stable LTI system M 
(input [f; w] € itm, output [e; v] € LAtk state x € £"), assuming the interconnection satisfies the IQC 
oa ([fs w3x]) > 0. 


f e 


—_—_—> ——s xP 


A 


The IQC o, > 0 is interpreted as one describing A. While a typical IQC model will have multiple IQC 
for every subsystem, the single IQC setup is general enough to formulate abstract statements relevant to 
feasibility optimization and postfeasibility analysis. 

The quadratic form oa : IRit+m+" 1s RR, as well as the real coefficient matrices A, B;, Ci, Di (where the 
pair (A, [B,, Bz]) is controllable) in the equations 


e Di Dy Ci |] f 
M: vi= D1 D2 C, wis x(0) € X (41.20) 
x By, Bo A x 


are given, while the feedback operator A and the (nonempty) set of initial conditions Vp C R” are not 
expected to be known in detail. 

The associated IQC model is given by the triplet (S,o.., A), where S = {q} is the behavioral model 
consisting of all signals q = [f; w; x], satisfying the Equation 41.20 and w = A(v) for some e, v, 


ox(Lfs ws x) = rf? —|Cix+ Dif + Dizw)? (41.21) 
is the parameterized “analysis objective” quadratic form, and 
A={o=coator: c>0, Q=Q’€R"*"} 


where o; = o7([f; w;x]) = 2x’Q(Ax + Bif + Bow) (“pure integrator” IQC). This is the set of quadratic 
forms o for which the IQC o & 0 on S is readily established. In particular, as mentioned in Section 41.2.1.5, 
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= Xq = Axo + Bor f + Bow as 


€ = CX + Dov + Dow 


. Vv = Co9%9 + Doaiv + Do,22W 


%_= A,X, + Biv + Beow 


FIGURE 41.3 Illustration of how the state-space extension contributes to the dynamics of M. The extended state 
vector does not affect the signals in the feedback loop (e, v, w). It is only used in the IQC that describes A. 


for continuous time signals of finite energy, the relation x = Ax + B,f + Bow implies o7([f; w; x]) > 0 for 
every Q = Q’. Moreover, o7 ([f; w;x]) > 0 when Q > 0 is positive semidefinite. 

The LTI system M in general includes a nominal LTI part of the system under investigation as well 
as a stable state-space extension used for the IQC modeling. As an example, consider Figure 41.3, where 
a linear system G is interconnected with A. By using an extended IQC description as in Figure 41.2 
we obtain an LTI model M with state vector x = [x9; xe]. The IQC oa (Lf; w;x]) > 0 on S that is used 
to describe A then depends on x not only via the output v = Co,2x9 + Do,21v + Do,22w but also via the 
extended state vector xe. This IQC also has a frequency domain representation. To see this, let 


|. re | = Pla. el — A)" EB Ball (41.22) 
T 2 


where the operation P = P{o, E} is defined in Section 41.3.1.5. Since M is assumed to be stable (i-e., A is 
Hurwitz), the I1QC o, > 0onS can be expressed in the frequency domain formatas [f; w]/T(jw) Lf; w] > 0, 
as long as Vp = {0}. 


41.3.2.2 Feasibility Optimization in Time and Frequency Domain 


Consider again the operator feedback setup in Section 41.3.2.1. The task of finding o9 € A such that 
0, > 09 with r as small as possible can be delegated to an appropriate optimization engine as a semidefinite 
program with decision variables c € Ry, Q= Q’ € R"*", r € Rand objective r > min. 

It is possible to find an equivalent frequency domain formulation. By the KYP Lemma (Theorem 41.1), 
the existence of Q = Q’ € R"*” such that the quadratic form 


0x(Lf3 w3x]) — con (f; w;x]) — o1(Lf; w,x]) = 0, 
is positive semidefinite is equivalent to the following condition*: 
rif? —|MiGo)Lf; wll? — cLfs wl PGo)Lf; wl > 0, Vifswle C'*™, weER (41.23) 


where M,(s) = [Mj1(s), Mi2(s)] = Ci(sI — A)~![B1, Bx] + [D1, D2], and I'(s) is as defined in Equa- 
tion 41.22. One can verify that there exists r,c € R; such that Equation 41.23 holds if and only if the 


* We use that A is Hurwitz and assume that (A, [B;, B2]) is controllable. 
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condition 


Py2(jw) < —€(P12(jo) Ti2(jo) + Mi2(jo)'Mi2(jw)) VoeR, (41.24) 


is satisfied for some € > 0. 


41.3.2.3 Need for Postfeasibility Analysis 


In general, the existence of c > 0 and Q = Q’ such that 0, > co, + 0; does not imply the complete IQC 
0x > 0. This could be due to the fact thato, > 0 buto, ¥ 0, or because Q = Q' is not positive semidefinite. 
On the other hand, it is possible to have o, ¥ 0 and Q # 0 while o, + 07 > 0. 

For example, consider the stable operator feedback setup from Section 41.3.2.1 given by state-space 
equations x} = —x, + f, x2 = —x%. + w,e=v=x, + x2 with X% = {0}: 


f u Vv w 


ji > oH + A > 


The conditional IQC o, > 0, where o, ([f3 w3 X13 X2]) = [x1 |? — 0.25|w|? (or, equivalently, |v — jori 


0.25|w|? 0) is satisfied for A = Ag as well as for A = Aj, where Ao(v) = 0 and Aj(v) = 2v. Also, 
existence of Q = Q’, c> 0, andr €R such that o, > co, +0; is guaranteed by the frequency domain 
condition (Equation 41.24), as in this case 


= 


teehee 3°); gree 
Lye — a a ii\S) = S 
0 —0.25 4 l+s 


Nevertheless, the complete IQC o,. > 0 holds for r > 1 when A = Ag, but the feedback interconnection 
is unstable, and the power gain from f to e equals infinity, when A = A}. 


41.3.3 Postfeasibility Analysis 


In this section, we briefly discuss conditions, to be imposed on a behavioral model S c C4 and quadratic 
forms 0,69: R4 + R, which guarantee that conditional IQC oo > 0 and the inequality 0, > o9 imply 
the complete IQC o,, > 0. 

Three approaches can be considered: minimal stability, the homotopy, and the minimax approaches. 
The later two are often convenient to apply in analysis of complex systems with many different compo- 
nents. This will be discussed in further detail in the next section. 

The minimal stability is a simple condition* to be imposed on og = 0, — coq, which guarantees that 
Q => 0 whenever o, > 07. Then o; > 0 and ox > 0 follows as long as it is known that o, > 0. 

In terms of the stable operator feedback setup from Section 41.3.2.1, minimal stability means exis- 
tence of real matrices K,,K2 such that Ax =A-+B,K,+B2Kz2 is stable (ie., Ax is Hurwitz), and 
Oq(K,x, Kyx, x) < 0 for all x € R”. Indeed, since og > 07, minimal stability implies QAx +AQ <0, 
hence Q > 0. Such Kj, K2 are usually easy to find when o, has a simple structure, as in the classical theory 
of absolute stability. 

For example, the derivation of the circle criterion in Section 41.2.2.3 uses the stable operator feedback 
setup with B, = 0, By = B, A(y) = o(y), and oa (Lf; w; x]) = 2c;w(Cx — w) satisfying on > 0. It employs 
a minimal stability argument, with K2 = 0, to show that Q > 0. The derivation of the Popov criterion 
in Section 41.2.2.4 can use a similar argument, with o([f; ws x]) = 2csw(Cx — w) + 2cpwC(Ax + Bw), 


* 


Invented and frequently used by V. Yakubovich. 
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when cp > 0 (and hence 2cpwC(Ax + Bw) > 0). When, cp < 0, 05 should be redefined as 
o3(Lfs w3 x]) = 2c,w(Cx — w) + 2cpwC(Ax + Bw) — 2cpx'C'C(Ax + Bw). 


Since 2cypwC(Ax + Bw) — 2cpx’C'’C(Ax + Bw) > 0 when cp < 0, the minimal stability argument can be 
used, with K, = C being the natural selection. Establishing that Ax = A+ BC is a Hurwitz matrix is the 
extra effort required in this case. 

The homotopy approach does not utilize information about o but requires the IQC o,, > 0 to be “strict” 
and the system to be parameter-dependent. It considers a family {S(t)} of behavioral models S(t) C £4 
depending continuously on parameter t € [0, 1] (so that S(0) is easy to analyze and S(1) is the true system 
of interest), and a quadratic form 0, : IR¢ +> R such that the conditional IQC o, & 0 is satisfied on S(t) 
for all t € [0, 1]. Conditions can then be derived under which, the complete IQC o,. > 0 on S(1) is implied 
by the complete IQC o, > 0 on S(0). 

The homotopy approach is particularly easy to use in the stable operator setup in Section 41.3.2.1. Then 
the homotopy is easy to construct due to the fact that all operators in the loop are stable, see Section 41.4 
and [3,8,9] for further details. The extension to more general situations is discussed in [11]. 

The “minimax” approach does not utilize information about o and does not require one to construct a 
homotopy of systems. Instead, it assumes that S = {q = [w; v; x]} is a feedback connection of a marginally 
stable “nominal” LTI system with inputs v, w and state x, and a causal stable feedback w = A(v). This 
includes the setup in Section 41.3.2.1 with the exception that the system matrix A could be marginally 
stable. The quadratic form oo is assumed to be such that, subject to the nominal LTI equations, the 
conditional IQC o,, > 0 is satisfied not only for the actual A, but also for A = 0, and is a convex constraint 
on w. These assumptions allow one to use the minimax theorem to prove the complete IQC o, > 0. A 
more elaborate discussion can be found in [7]. 


41.4 Application of IQC Analysis 


The framework discussed in the previous section is a flexible and versatile approach for systems analysis 
that can be used in many contexts. In this section we illustrate how IQC can be used in analysis of 
systems consisting of a nominal part interconnected with a number of more challenging components Ax, 
k =1,...,N; see Figure 41.4. We restrict our attention to the case where the components A; are stable 
causal operators and the nominal part is LT] and stable with a transfer function representation G(s). It is 
assumed that G(s) has a state-space realization as in Figure 41.3. The purpose of the analysis is as before 
to verify an energy gain from input f to output e. 

One of the main advantages of IQC is that components of very different nature can be characterized 
in a unified way. The analysis framework can therefore be implemented as a software package consisting 
of a library of component descriptions, a compiler that assembles the IQC descriptions, and finally, an 
optimization engine to solve the resulting feasibility optimization problem. One such prototype, IQCB, is 
implemented in MATLAB; interested readers are referred to [6] for more details. 

In this section we briefly describe the IQC analysis flow and apply it to a simple example. A brief 
“library” containing the two IQC descriptions used in the example is also included. 


— Nominal system -— 


“| vy ws ON 
Al see AN 


FIGURE 41.4 Illustration of a system consisting of a nominal part and complicating elements Ay, k = 1,...,N. 
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41.4.1 The IQC Analysis Flow 


The IQC framework applied to the system in Figure 41.4 involves the following three steps. 
IQC modeling: The first step is to find IQC descriptions for the A,. For this purpose, we consider 
behavioral models that usually contain a stable LTI extension 


Sa, = (9k = [Wks Ves Xek] : We = Ak(VE)s Xek = AckXek + Beki vk + Bek2wx} 


and set of quadratic forms A, = {o,} such that o,(q,) > 0 on Sa, for each 0, € Ax. These IQC descrip- 
tions can be assembled to obtain a complete IQC model of the form (S,o,, A), where S = {q = [f; w; x]} 
is a behavioral model for the whole interconnected system, consisting of signals satisfying a state-space 
equation of the form in Equation 41.20 with x = [xo;%e1,...,%en], v=[v13...; vw], w=[w1;..-3 wn] 
and w, = Ax(v,). The quadratic form o,, is the analysis objective of the form in Equation 41.21 and 


A=[o= Dace tor:oc€ Ab a= 0 


is a set of quadratic forms for which o > 0 on S. 
Feasibility analysis: The feasibility analysis can be formulated as the semidefinite program 


minimize r subject to 0, — a CkKIOk — Or > 0; cE =O (41.25) 


for some fixed ox, € Ay, k= 1,...,n. Since we are using a strict inequality feasibility of this optimization 
problem implies that the IQC o,, > 0 is strict, which is necessary in order to use the homotopy approach 
for postfeasibility analysis. 

This optimization problem can sometimes be of a very large dimension, which makes solving the 
problem numerically a challenge. In such cases, in order to improve the computational efficiency, it 
becomes crucial to explore and exploit special structures of the optimization problem. One idea, proposed 
and developed in [12], is to solve the dual formulation of the semidefinite program in Equation 41.25. 
The number of decision variables to be optimized in the dual formulation is usually significantly smaller 
than those of Equation 41.25; thus the computational complexity of the dual problem is substantially 
reduced. Another idea is to reformulate Equation 41.25 into an equivalent semi-infinite form and solve 
the semi-infinite problem using specialized algorithms. Along this line of thought, several algorithms 
have been proposed, developed, and tested. See [4,5] for more details. 

Postfeasibility analysis: By assembling the IQC descriptions for the A, we arrive at the stable operator 
setup in Section 41.3.2.1 with A(v) = [A1(v1);...; Aw(vn)]. Application of the homotopy argument to 
the stable operator feedback setup is generally straightforward. It is sufficient to use a parametrization of 
the form 


S(t) ={Lfswix] e c+" : w = A,(v); Equation 41.20 is satisfied}, 
where A, is a continuous parametrization of A such that 


There exists y > 0 such that ||A,, — Az, |] < ylti — tI, for 1, t2 € [0, 1] 
Aj=A 

S(0) is a stable system, that is, o, > 0 on S(0) 

oa > 00n S(t), for t € [0,1] 


PTO oS 


The simple parametrization A, = tA is often sufficient. 

It is sometimes also possible to apply the minimax approach by exploiting convexity properties of the 
quadratic forms that define the IQC. The later approach is particularly easy to apply in the frequency 
domain formulation described next. This formulation is in line with how IQC analysis was presented 
in [8]. 
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We have seen in the previous sections the IQC can be formulated in both time and frequency 
domain. Each of the IQC descriptions considered above is equivalent to a frequency domain IQC 
[vis Wel’ TH. jo) v3 we] & 0, where 


, = Ploa, (sI — Ack)‘ [Beki> Bexz]}. 
In fact, we may use a convex set of IQC obtained as 


Ta, = De Ck Tk, * Ck 2 o| , 


where P{o,,,.> (sI — Ack) ‘[Bek1s Beka] for some fixed OAa,, € Ax. The three-step procedure of IQC anal- 
ysis goes as follows. 

IQC modelling: The model of the linear part of the system has the form Sj, = {[f; w; ev] : [ev] = 
G[f;w]}, where G has block row structure G = [Go; G1; ...; Gy] with dimensions consistent with the 
output signals [e; v1; ...; vw]. The complete system has the behavioral model 


S={q=[fswsesv]:q € Shins we = Ax(vy)}. 


For this, we have the analysis objective 
2 2 
rlf|“ —lel“ > 0 


and the IQC descriptions 
[ves wel Tx jo) [vp wele>0, Whe TI,,- 


Feasibility analysis: The feasibility criterion can be formulated as a frequency domain constraint 
N 
rif? —IGo(jo)Lfs wll? > STG Go)Lfs wll Te Gol Ge io)Lfs wl] 
k=1 


for all w € RU {oo}, [f; w] € C. This can equivalently be rewritten 


1 0/0 0 
/ / 
G 0 Wy 0 TI) G . 
0 41.26 
id oO. l=r 0 H Gays ee) 
O® Te ||co: Siig 


for all w € RU {oo}, where Ty = diag(I1),4,..-, Ty41), k, 2 = 1,2. The feasibility optimization can thus 
be formulated as 


minimize r subject to Equation 41.26; I, € ITa,. 


Post-feasibility analysis: The minimax approach applies to this setup if I11,(jw) > 0 and IIz2(jw) < 0 
for all w € R. Otherwise, the homotopy approach applies under the same assumptions as discussed above. 

The above analysis expression can be arrived at more efficiently as we will see in the example presented 
in Section 41.4.3. 


41.4.2 IQC Library 


There exists a large number of IQC established for common nonlinear, time-varying, uncertain, and 
distributed elements of dynamical system models. Here we present two that will be used below. 
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41.4.2.1 Monotonic Odd Nonlinearity 


Let ) : R > R be an odd function such that (0) = 0 and b € [0,k] for some constant k. Let Smon = 
{[w;v]: v € L, w(t) = o(v(t))} be the behavioral model describing the mapping v(t) > (v(t), and 


(oe) 


Smon,e = {tvs :[w;v] ES, f(t) =I, 


—0oO 


h(t — t)w(t) ar| : 


[oe 


be an extended model for Smon, where h is any function which satisfies d |h(t)| dt < 1; that is, the 
—0o 


L,-norm of h is bounded by 1. It can be shown that Smon,e satisfies IQC oz¢ > 0, where 
Oz ([w3 v3 f]) = a(v — w/k)(w —f), 


and a > 0 is any nonnegative real number. This IQC is derived from a classical paper by G. Zames and 
P. Falb [16], and sometimes referred to as the “Zames-Falb” IQC. Note that, although any h whose L;- 
norm is bounded by 1 can be chosen to form the extended model Spon, one would usually select an h 
which is the impulse response of some rational transfer function in order to take advantage of utilizing 
convex optimization tools in the feasibility analysis step. 

In the case when the function $ is not odd but satisfies the other properties above, the IQC oz > 0 
is still valid as long as the additional condition h(t) > 0, Vt holds. For example, the IQC presented in 
Section 41.2.1.6 corresponds to selecting h(t) = 2(e~! — e~2') for t > 0 and h(t) = 0 when t < 0. It can 
be readily verified that h is the impulse response of transfer function H(s) = 2/(s* + 3s+ 2), and the 
L,-norm of h is equal to 1. The corresponding extended model Smon,e can be equivalently expressed as 


Smon,e = {Iwsvsfsfl :[w;v] €S,f +3f + 2f =2w,f(0) =f(0) =0| > 
which satisfies IQC 2(v — w)(w—f) > 0. 


41.4.2.2 LTI Unmodeled Dynamics 


Let A: £” + £™ be a LTI bounded operator with the L2-induced gain being less than or equal to 1. Let 
Sud = {[wsv] : v € £", w = Av} be the behavioral model describing the LTI transformation v+> Av. It 
can be shown that Syq satisfies the frequency weighted IQC oyg > 0, where 


of! (Lw; v]) = a(jo)(|v|? — |w|?), 


for every rational function « € RL!*! which is bounded and nonnegative on the imaginary axis. This IQC 
can be proven by observing that the Fourier transforms ¥, w of finite energy [w; v] € Sua satisfy 


W(jw) = AGw)?(jo), VoeR, 


where A(s) is the representation of A in the Laplace domain. Since sup,, 6 (Ac jw)) < 1, where o(M) 


denotes the maximal singular value of M, we conclude that 
Goll? < PG), Yo eR, 
and hence the IQC. 


41.4.3 Example 


Figure 41.5 shows a model of a servo system consisting of a nominal plant P, a controller K, a standard 
saturation nonlinearity, and a dynamic uncertainty A, which represents unmodeled flexible modes. It is 
assumed that the energy gain of A is upper bounded by x. The analysis objective is to verify whether the 
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FIGURE 41.5 Feedback system with saturation and unmodeled dynamics. 


energy gain from the input disturbance f to the output e is less than ,/r. In this case we use quadratic 
form o, = rle|? — |f|? as the analysis objective, the IQC description 


ay - 2Re[(v1 — w1)(1 — H(jw)) wi] > 0 (41.27) 
for the saturation nonlinearity, and the IQC description 
02(jw)(k?|v2|? — |w2|7) & 0 (41.28) 


for the dynamic uncertainty, where a; > 0, a2(jw) > 0, and H is a transfer function whose impulse 
response function has L;-norm less than one. The frequency domain formulation of the analysis condition 
can directly be formulated as 


r|P(jw)(f +1) + wal? — fl? > a2(jw)(k?|PGiw)(f + wi)? — |w2l) 


; : . (41.29) 
+ 20 Rel (—K(jo)(w2 + P(jo)(w1 +f)) — v1) 1 — H(jo)) wi] 


for all w € RU {oo} and f, w1, w2 € C. It is easy to see that the postfeasibilty conditions discussed above 
are satisfied. 

In the previous section, we mentioned that the IQC analysis framework is implemented in MATLAB. 
The toolbox, named IQCf, consists of a parser and a library of IQC descriptions for various kind of 
uncertain/nonlinear operators. The toolbox allows the users to perform IQC analysis in the MATLAB 
environment, using syntax similar to the language of MATLAB. The parser interprets the IQC models, 
descriptions, and the analysis objective that users input to the computer, defines the corresponding 
optimization problem for feasibility analysis, and calls for an optimization engine (an LMI solver if the 
problem is formulated as a set of finite dimensional LMIs) to solve the optimization problem. In the 
following, we briefly illustrate how to use IQCf to perform the energy gain analysis described above. 
For the sake of illustration, let the plant P and the controller K be modelled by the following transfer 
functions: 


—s+l1 1.92s — 3.471 


P(s) = —————_., K(s)= : 
() s*+0.01s+1 (s) s* + 4.628s + 11.02 


The MATLAB codes for calculating an upper bound of the energy gain are given below. 


s = tf£([1,0],1); %(1) 
P = (-s4+1)/(s*s+0.01*s+1) ; %(2) 
K = (1.92*s-3.471)/(s*s+4.628*S+11.02) ; %(3) 
abst_init_iqe %(4) 
£ = signal; %(5) 
wl = signal; %(6) 
w2 = signal; %(7) 
v2 = Px(f+wl1); %(8) 
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e = w2+v2; % (9) 

vil = Kx(-1)xe; %(10) 
wl == iqc_monotonic(v1,1,1,1); $(11) 
w2 == iqc_ltiunmod(v2,1,1,0.01); $(12) 
igqce_gain_tbx(f,e) $(13) 


Command lines (1) through (3) define the transfer functions P(s) and K(s). The function/operations used 
in these command lines are native to MATLAB and not a part of the IQCB toolbox. Command lines 
(4) through (13), on the other hand, utilize functions and operations which are parts of IQCB and will 
not work if the toolbox is not installed. Command line (4) initiates the environment for IQC analysis 
under MATLAB. Command lines (5) through (10) define the linear part of the system in Figure 41.5; 
that is, the corresponding Sj, for this system. In this example, the linear part of the system has three 
“external inputs” f, wi, and w2, which are defined by command lines (5), (6), and (7), respectively. 
The “internal signals” v2, e, and vj, are related to f, w;, and wz by command lines (8) through (10). 
Command line (11) defines the IQC model for the saturation nonlinearity, which is embedded as a 
monotonic odd nonlinearity. The IQC relation in Equation 41.27 for signals w; and v is defined in the 
function iqgc_monotonic.m. Here the corresponding H(jw) is 1/(jw + 1). Likewise, Command line 
(12) defines the IQC model for the dynamic uncertainty A, and the IQC relation in Equation 41.28 for 
signals wz and v2 is defined in the function igc_ltiunmod.m. The corresponding a2(jw) has the 
form x/(ja-+ 1), where parameter x is a positive real number. We also note that for this illustration, we 
assume the energy gain of A to be less than 0.01. Finally, command line (13) executes the IQC parser, 
which collects the information defined by command lines (1) through (12), formulates the corresponding 
optimization problem in Equation 41.29 as LMIs, and call the generic LMI solver provided by MATLAB 
to find the least upper bound of energy gain. In this example, the bound found by the optimization engine 
is around 142.9. 


41.5 Historical Remarks and References 


The use of IQC in systems analysis has a long history. The term was, to our knowledge, introduced by 
V.A. Yakubovich in the 1960s in a sequence of works where advanced sector and Popov criteria were 
developed using IQC. Yakubovich continued to develop this approach and a recent survey can be found 
in [15]. 

Several related and equally important directions were initiated in the 1960s and 1970s. The theory 
of dissipative dynamical systems was introduced by J.C. Willems with a focus on the use of dissipation 
inequalities and positive definite storage functions as a natural way to quantify energy dissipation in 
physical systems. See, for example, the recent survey in [14]. Another approach to stability analysis 
was the input-output theory developed by Zames, Sandberg, and others. The most powerful results are 
obtained by using the multipliers, an idea, which to our knowledge was introduced by Brockett and 
Willems [1] and further developed to allow the use of noncausal multipliers by Zames and Falb [16]. 
Extensive accounts of the input-output theory can be found in the books [2,13]. 

Later in the 1980s and 1990s the attention turned to more complex systems with structured uncer- 
tainties. The development of efficient algorithms and software for convex optimization motivated the 
computational frameworks for robustness analysis in, for example, [10]. However, as more complex 
uncertainty structures were considered it became evident that the input-output theory required certain 
factorization conditions on the multipliers, which complicated the analysis. This was overcome by using 
IQC defined in the frequency domain as in presented [8]. This tutorial paper presents the IQC approach 
closer to the way it is implemented in the custom-made MATLAB toolbox IQCf [6]. It provides a general 
tool for analysis of complex systems containing heterogeneous components. A previous tutorial on IQC 
can be found in [3] and further perspectives are given in [9,11]. 
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42.1 Introduction 


In this chapter, we study motion planning and control for systems subject to nonholonomic and underac- 
tuation constraints. All such systems can be written in the form 


k=f(x)+) uigi(x), uieR, (42.1) 


i=1 


where f (x) is a drift vector field, g;(x) are linearly independent control vector fields, and u = [u,..., Um]! 
is the control. Unless otherwise specified, we work in local coordinates and treat the state as x € M = R?. 
While many nonlinear systems can be written in the control-affine form (Equation 42.1), we are partic- 
ularly interested in first-order kinematic systems and second-order dynamic mechanical systems. In the 
case of kinematic systems, the state x is simply the configuration q € Q = R" (i.e., p = n), the control vector 
fields are velocity vector fields, and the controls u are speeds. For second-order mechanical systems, the 
state is x = (q, q) in the tangent bundle TQ = R2", the control vector fields encode acceleration directions, 
and the controls u are generalized forces. For both first-order kinematic and second-order mechanical 
systems, we are interested in the case of fewer control inputs than configuration variables, m <n. 
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A classical nonholonomic constraint is a velocity constraint that cannot be integrated to yield an 
equivalent configuration constraint. A common kind of velocity constraint is a Pfaffian constraint of the 
form 


a(q)q = 0. (42.2) 


If there exists a function h(q) such that 0h/0q = (q)a(q) for a suitable function |1(q), then the Pfaffian 
constraint is not nonholonomic, but is the derivative of a holonomic constraint. A holonomic constraint 
reduces the dimension of configuration space. If there is no equivalent holonomic constraint, then the 
constraint is nonholonomic and reduces the dimension of the feasible velocities, but does not reduce the 
dimension of the reachable configuration space. 

Our interest here is in nonintegrable Pfaffian constraints of the form A(q)q = 0, where k velocity 
constraints are encoded in the n x k matrix A(q). Such nonholonomic constraints arise from rolling 
without slip, and may be implicit in some conservation laws, such as conservation of angular momentum. 
The constraints define a distribution D in the tangent bundle TQ, and Dj denotes the subspace of feasible 
velocities at each q. 

While kinematic systems may be subject to nonholonomic velocity constraints, second-order systems 
may be subject to both velocity and acceleration constraints. Acceleration constraints arise due to the fact 
that the system is underactuated—it has fewer control inputs than configuration variables (m < n).* If 
acceleration constraints cannot be integrated to yield equivalent velocity constraints, we refer to them as 
second-order nonholonomic constraints. 

Thus the systems of interest in this chapter are kinematic systems with velocity constraints, and second- 
order mechanical systems with velocity and/or acceleration constraints. All of the systems we consider 
can be expressed in the form (Equation 42.1). 

Control of nonholonomic and underactuated systems is challenging, as we experience every time we 
try to parallel park a car. This chapter focuses on the following challenges: 


¢ Evaluating controllability. Given a description of the system in the form (Equation 42.1), Lie alge- 
braic tests can be used to study the reachable state space. Some controllability concepts are specific 
to second-order mechanical systems and allow simplified tests. 

¢ Feedback stabilization. A classic result due to Brockett prompts the search for discontinuous or 
time-varying feedback control laws. Another strategy is to derive motion planners and to use 
feedback control to stabilize planned trajectories. 

« Motion planning. The motion planning problem is to find an efficient algorithm that gives an 
open-loop control steering the system from a start state to a goal state. 


In the next section we provide examples of nonholonomic and underactuated systems, and the remain- 
ing sections address the issues above. General references for further reading include [2,5,11,13,18]. 


42.2 Notation and Examples 


We can classify example systems according to whether they are first-order kinematic systems or second- 
order mechanical systems. For first-order kinematic systems, the system state x is simply the configuration 


* At least two definitions of “underactuated” are possible: (1) the number of control inputs m is less than the number of 


configuration variables n, as adopted in this chapter or (2) m is less than the dimension of Dg, the number of linearly 
independent instantaneously feasible velocity directions. For example, a simple model of a car has n = 3 (the position and 
heading direction of the car in the plane) and m = 2 controls (the forward and turning speeds). The car is underactuated 
by the first definition, but not by the second—the no-slip constraint of the car means that the control vector fields span 
the space of feasible velocities. We could modify our definition of “underactuated” to be that m is less than the number 
of “degrees of freedom,” but this also does not settle the matter: some authors equate “degrees of freedom” to n, while 
others equate it to the dimension of Dg. In any case, once the system is converted to the form (Equation 42.1), there is no 
ambiguity, and m < n for all systems we study. 
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FIGURE 42.1 The geometry and configuration coordinates for the vertical rolling disk. 


q, and k = n—m > 1, that is, the number of velocity constraints is equal to the dimension of the con- 
figuration space minus the dimension of the control space. For second-order mechanical systems, there 
are three possibilities: k = 0 (and n— m > 1), that is, there are no nonholonomic velocity constraints; 
k =n—m > 1, that is, the actuators can apply a generalized force in each unconstrained velocity direc- 
tion; and n—m>k > 1, that is, there is one or more unconstrained velocity direction which has no 
associated actuator. We give examples of each of these cases below. More detail on these examples can be 
found in [2,11-13,21]. 


42.2.1 Kinematic Systems 
Example 42.1: The Vertical Rolling Disk 


The configuration of a disk of radius R rolling on a horizontal plane can be parameterized by 
q=lx,y,v, ol’, as shown in Figure 42.1. The k = 2 nonholonomic rolling constraints can be written 
A(q)g = 0, where 
1 0 —Rcosd O 
aS E 1 -Rsing AR 23) 


The m =2 controls are the wheel rolling velocity u; = y and the heading rate of change u2 = ¢, 
with corresponding control vector fields g1(q) = [Rcos 6, R sino, 1, ol! and g2(q) = [0, 0, 0, 1)’ .There 
is no drift field, f(g) = 0. 

It is easy to confirm that the rolling disk can reach any configuration in Q (see Section 42.3). 
Therefore, the constraints A(q)qg = 0 are indeed nonholonomic—they do not restrict the reachable 
configuration space. If we add one more velocity constraint, u2 = = 0, however, we see that 
the three constraints taken together can be integrated to obtain three holonomic constraints. The 
reachable configuration space has been reduced from four-dimensional to one-dimensional integral 
manifold of the vector field g;; the disk can only roll back and forth on a line, and its position on 
the line, together with the initial configuration, determines wy. While the reduction in the reachable 
space is obvious in this example, and indeed in any drift-free example with a single control vector 
field, in general the integrability of constraints can be evaluated by considering the Lie algebra of 
the system vector fields (Section 42.3). 


Example 42.2: The Heisenberg System (Nonholonomic Integrator) 


Consider the vertical disk example, but eliminating W from the representation of the configuration. 
The system can be written 


x = vR cos o, 
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y=vRsino, 
b=, 


where the forward velocity and heading velocity controls are v and w, respectively. We can define a 
change of coordinates F() 


x1 x 
x2| =F(o)}y |, 
Zz 
where 
0 0 1 
F(o) = cos sind oO}, 
dcosd—2sind dsind+2cosh O 


and a nonsingular state-dependent transformation of the controls 


Uy, = 0, 
Z X1X2 
u2 =Rv+ (5 - a) F 
. 2 2 
yielding the system 
X2 = U2, (42.5) 
Z=xX U2 —X2U1 (42.6) 


defined by the control vector fields g; = [1,0, —x9]! and g2 = [0, 1x4]. This system is called the 
Heisenberg system or nonholonomic integrator [8,9]. The importance of this system is that the asso- 
ciated three-dimensional Lie algebra is the Heisenberg algebra appearing in quantum mechanics. 
Generalizations can be used to model a number of other important systems. 


Example 42.3: Wheeled Mobile Robots 


Kinematic wheeled mobile robots are perhaps the best-known examples of nonholonomic sys- 
tems. Some of the most common wheeled mobile robots can be described by the Heisenberg 
system. We denote the configuration of the robot in a horizontal plane as q= [x,y, ol’, where 
(x,y) is the location of a reference point halfway between two wheels of the robot, and ¢ is 
the orientation of the robot (Figure 42.2). For a differential-drive robot, the two wheels are not 
steered but their speeds can be controlled independently. For a car-like robot, the reference point 
is between the unsteered rear wheels. In both cases, the system is described by the vector fields 
gi =[cos9,sing, o}! and g2 = [0,0, 1]7 with controls u, = V, the forward velocity, and u2 = w, the 
angular velocity. 

A number of different control sets have been studied for wheeled mobile robots. Four sets are 
shown in Figure 42.2, corresponding to a car with a limited forward-reverse speed and limited turning 
radius; a differential-drive robot with limited wheel speeds; a “Reeds-Shepp” car, which is a car that 
can only take forward-reverse velocities in the set {—Vmax, Vmax}; and a forward-only “Dubins” car 
that has v fixed at Vmax. For simplicity, in this chapter, we treat control sets as R™, but some results 
also apply to restricted control sets as in Figure 42.2. 


42.2.2 Dynamic Mechanical Systems 
42.2.2.1 Systems with Velocity Constraints 


To describe a constrained mechanical system, we need the following elements: a symmetric 
positive-definite inertia tensor M(q) € R"*” and a potential energy U(q) € R defining the Lagrangian 


Control of Nonholonomic and Underactuated Systems 42-5 


O, OO, 
v v 
Car Diff-drive robot 
M4 M4 
| | | | 
rie i 
Reeds—Shepp car Dubins car 


FIGURE 42.2 Configuration coordinates of a wheeled mobile robot, and four different control sets in the space of 
forward velocities v and turning rates w. 


L(q,q) = +q'M(q)q — U(q); a set of controls U/, which we take to be R” for simplicity; and an input 
tensor T(q) € R"*™ indicating how a control u acts on the generalized coordinates q. If the system is 
subject to Pfaffian velocity constraints, then the k constraints are written A(q)q = 0, where A(q) € R**", 
and these constraints result in a set of constraint forces A’ (q)d, where  € R*. 

We define the n° Christoffel symbols of M(q) 


1 (Soa _ Omix(q) ay 


2 O4k : 04; Ogi 


T, (q )= 
where mjj(q) is the (i, j)th element of M(q). Then Lagrange’s equations yield the dynamics 


M(q)q + (qq) = T(q)u+ A" (q)r, (42.7) 
A(q)q = 0, (42.8) 


where b(q, 4) = q'T'(q)q+0U(q)/0q, and the ith element of the n-vector g'T(q)q of Coriolis and 
centripetal terms is defined as q’ I'(q)q. 
For Equation 42.8 to be satisfied at all times, we must also have 


A(q)q+ A(q)q = 0, (42.9) 
allowing us to solve for the the Lagrange multipliers ). Dropping the dependence on q, 
» = (AM1A‘)-1(AM71(b— Tu) — AQ). (42.10) 


Plugging Equation 42.10 into Equation 42.7 to eliminate the Lagrange multipliers, we obtain the n 
dynamic equations 
g=M~!(A"(AM~!A")~!Ag—P+b+P*Tu), (42.11) 
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where 

Pt =1,—A'(AM~'A™) 1AM"! (42.12) 
is an n x n matrix of rank n—k that orthogonally projects (by the kinetic energy metric) generalized 
forces to the components that do work on the system, and Z,, is the n x n identity matrix. Equation 42.11 
can be expressed more compactly as 


PY @(M@a+ 6) =P-@T@u 
or 
P(q)(4+M*(q)b(q, 4) = P(q)M~*(q)T(q) u, (42.13) 
Y(q) 
where P = M~!P+M is an n x n matrix of rank n — k that orthogonally projects motions to those that 
satisfy the velocity constraints. P and P+ are orthogonal by the kinetic energy metric. We call the columns 
of Y(q) input vector fields, discussed in Section 42.3.2.2. 

Using Equation 42.11, we could now write the 2n-dimensional drift and control vector fields of 
Equation 42.1. IfEquation 42.11 is integrated exactly, the system state evolves on the (2n — k)-dimensional 
distribution D defined by the k nonholonomic velocity constraints. Note, however, that there are only 
n— k independent equations in Equations 42.11 and 42.13, and when we examine controllability of the 
system using Lie brackets, we use Taylor expansions of the system vector fields. Because these only 
approximate the vector fields, our controllability analysis may incorrectly conclude that the system can 
escape D to other points in TQ. See Section 42.3.2.2 for controllability tests based on the dynamics 
(Equation 42.13). 

As an alternative, we can use the nonholonomic Pfaffian constraints to eliminate k velocity state 
variables and arrive at a reduced set of equations on the (2n — k)-dimensional distribution D. Reordering 


the configuration variables and labeling them q = [q!, qi ]’, we can write the Pfaffian constraints as 
A(q)q=[A1(q) A2(q)] ie =0, (42.14) 
~—S— ~——_— ——’ q2 
kxn kxn—-k — kxk 


where A2(q) is invertible. The k-dependent velocities qz can be calculated from the n — k-independent 
velocities q, by 
42 = D(q)qr = —Az' (Q)A1 (4) an. 


Ts 
C(q) = Fal , 


q=C(q)q, (42.15) 
q@= C(q)qi + C(@)in. (42.16) 


Substituting Equations 42.15 and 42.16 into Equation 42.7 and premultiplying both sides by C"(q), we 
obtain 


Defining the n x (n — k) matrix 


we obtain 


CQ M@CQa = C@[TMu— bg. C4) — M@C@ai}- 
Premultiplying each side by the inverse of the full rank (n—k)x(n—k) matrix M (q)= 
C(q)M(q)C(q), and dropping the dependence on q, we obtain 


ey 25 Py ea Py ns 
qa =M CC Tu—M CC’ (b+ MCq). (42.17) 
Hq) e(qaqu) 


We have reduced the n dynamic equations with constraint forces in Equation 42.7 to n — k equations for 


qi. Defining the state variables x; = qi, x2 = q2, x3 = 41, we can write the 2n — k state equations in the 
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form (Equation 42.1): 


xy x3 0 
x= | x | = | D(x, x2)x3 | + 0 u. (42.18) 
3 —c(x) A(x, x2) 
a ae 
f(x) G(x1,x2) 


The m columns of G(x), x2) € R@"-"*™ are the control vector fields gi(x) in Equation 42.1. 


42.2.2.2 Systems without Velocity Constraints 


In the case of no velocity constraints (k = 0), we can simplify the equations by first partitioning 
the configuration variables into those that are acted upon by the inputs and those that are not, 
q’ =([q),qi1,qi € R", q € R"-™.* Lagrange’s equations can then be written 


Mi1(q)q1 + Mi2(q)g2 + b1(q, gq) = B(q)u, (42.19) 
M1 (q)q1 + Mo2(q)q2 + b2(q, Gq) = 0, (42.20) 


where B(q) € R*” is invertible for all q < Q, b\(q,q) € R™, b2(q,q) € R"-™, and Mj(q), i,j = 1,2, are 
components of M(q). We may solve for q2 as 


go = —M33 (q)[Mo1(q)qi + b2(q, 1 
and substitute into Equation 42.19 to obtain 
M(q)an + b(q, 4) = B(q)u, 


where 


M(q) = Mii(q) — Mi2(q)M5,' (q)Ma1 (q); 
b(q, 4) = b1(4,. 4) — Mi2(q)M3,' (q)b2(q, @)- 


Consequently, using the partial feedback linearizing controller 


u = B'(q)[M(q)v + bg, 1, 


Equations 42.19 and 42.20 can be rewritten as 


ql=%, (42.21) 
g =J(g)qi t+ RQ), (42.22) 
where 
J(q) = —My,'(q)Mn1(q); 
R(q. q) = —M3; (q)b2(q Q). 


Equations 42.21 and 42.22 have a special triangular or cascade form that appropriately captures 
the important attributes of underactuated mechanical systems. Equation 42.21 defines the linearized 
dynamics of the m completely actuated degrees of freedom. Equation 42.22 defines the dynamics of the 
n—m unactuated degrees of freedom; these are expressed in terms of equalities involving the generalized 
accelerations. If these latter relations do not admit any nontrivial integral, that is, any nonconstant smooth 


* Tt may be necessary to perform a coordinate transformation to obtain this form. 
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function h(q, q, t) such that dh/dt = 0 along the solutions, then these relations may be interpreted as n — m 
completely nonintegrable acceleration constraints (or second-order nonholonomic constraints). As will 
be seen in the subsequent development, controllability and stabilizability properties of underactuated 
mechanical systems are closely related to this property. Hence, it is crucial to identify underactuated 
mechanical systems where the acceleration relations defined by Equation 42.22 are completely noninte- 
grable. 

Equations 42.21 and 42.22 can be expressed in the form (Equation 42.1) by defining the state variables 


M=Q, 2=qQr, %3=Gi, X=Q. 


Then the state equations are given by 


x1 = x3, (42,23) 
X2 = x4, (42.24) 
*3=V; (42.25) 
x4 = J (x1, x2)v + R(x1, x2, 3, X4). (42.26) 


Equations 42.23 through 42.26 define a drift vector field f(x) = [2,5] 07 R Gao aa, xa and m 
control vector fields gj(x) = [o! of, ae ve (x1X2)]", where e; denotes the ith standard basis vector in R™ 
and J;(x1, x2) denotes the ith column of the matrix-valued function J (x1, x2). 


42.2.2.3 Examples 


Example 42.4: The Dynamic Rolling Disk 


Consider the rolling disk of Figure 42.1, but now with rolling and steering torques as inputs, instead of 
angular velocities. In this example, n = 4,m = 2,andk = 2, and therefore k = n— m= 2—there is an 
actuator for each unconstrained velocity direction. Following the development in Equations 42.14 
through 42.18, we reorder the configuration variables from the kinematic case to q = fatally, 
where gq, =[W,o] and q7 = Ixy]. The system is described by the reordered rolling constraints 
from Equation 42.3, A;(q)g; +A2(q)g2 = 0, where 


_|-Rcosh 0 _ {1 0 
aila=| Theme of aca=[) ae 


| c 0 
0 0 1 
ol’ T(q)= 0 ol 
m lo 0 


where m is the mass of the disk, J is the inertia of the disk about an axis perpendicular to the face of 
the disk and through the disk’s center, and / is the inertia of the disk about a vertical axis through its 
center. The disk has zero potential energy, U(q) = 0. Setting x7 = q1, X2 = q2, X3 = q1, the drift and 
control vector fields in Equation 42.18 on the six-dimensional distribution D are described by 


and 


F 
mq) =| 6 


0 


oo~o 
o3 00 


1 
R 0 0 | sama? 0| 
Dixy aa) = [Pose ah w= [5]. H(x1,X2) = oe 1} 
| 
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FIGURE 42.3 The snakeboard model. 
Example 42.5: The Snakeboard 


The snakeboard is a commercial toy whose concept is derived from the well-known skateboard. It is 
composed of two steerable wheeled platforms joined by a coupling bar, and the rider propels itself 
forward without touching the ground by steering the wheels and twisting it body back and forth. 
A simple model of the snakeboard is shown in Figure 42 3. Here a momentum rotor simulates the 
rider by spinning back and forth, and by conservation of angular momentum about the rotation 
center chosen by the wheels, the snakeboard body moves. 

Our simple model of the snakeboard has n = 5, m= 2, and k = 2, and therefore n— m > k = 2— 
there is no actuator for each unconstrained velocity direction. Let the configuration of the snakeboard 
be represented by q = [6, w, > x yl", where (x, y) represents the Cartesian position of the center of 
the snakeboard coupler, 6 is its angle, and W and ¢ are the angle of the rotor and the steering angle 
of the wheels, respectively, expressed in the body frame. The inertia matrix for the simplified model 
of the snakeboard is given by 


Pedi! ie OOo 1 
i. 00 | 
M= 0 Oo de 0 OA, 
0 0 0m 0 
0 0 0 0m 


where m is the total mass of the snakeboard, | is the inertia of the coupler about its center of mass, 
I, is the rotor inertia, and Vy is the inertia of each set of wheels about its pivot point. (Note that 
because the inertia matrix is invariant to the configuration, the Christoffel symbols are zero.) The 
system is subject to two control inputs: a torque u; that controls the rotor angle Wy, and a torque u2 
controlling the steering angle o. Therefore, T(q) can be written 


0 0 
[ro] 
T(q)= | 0 1 | ; 
0 0 
lo 0 
The wheels are assumed to roll without lateral slipping, and the wheel angle chooses a rotation 


center along a line perpendicular to the body of the snakeboard and through its center. The no-slip 
constraints can be written in the form A(q)g = 0, where 


A(q) = —fLeosOcoshd O O. sing 0 
~~ |—£sinOcosh 0 O O_- singd|’ 


Following the development in Equations 42.14 through 42.18, we could choose qi = [6, v, >], 
qQ= Ixy], and A2(q) to be the right 2 x 2 submatrix of A(q). This yields the state equations 42.18 on 
the eight-dimensional distribution D away from states where  € {0, 11}, where A>(q) is noninvertible. 
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FIGURE 42.4 The 3R robot with three revolute joints. 


Example 42.6: The 3R Robot 


A 3R robot manipulator consists of three revolute joints (Figure 42.4). We consider such a robot 
operating in a horizontal plane with a passive joint—one joint lacks an actuator. This is an example 
of an underactuated system with no velocity constraints (k = 0). 

This system is described by q = [61, 92, 63], zero potential energy, and 


my m2 «M43 
M(q)=|™m21 M22. M23 |, 
m3; M3233 


where 
- 2 24.0) 2,,2,,2 
my =h+h+l3+mry t+mo(ly +5) +m3(L7 +5 +13) + 2moLi12¢2 
+ 2m3(L1L2c2 + Lor3c3 + L113¢23), 

m2=M21=2 +/+ m2(r5 +Ly1Q¢2)+ m3(L5 + re +LyLgc2 + 2Lgr3c3 + L113¢73, 

m3 = M31 =/3 + m3(r3 + Lor3c3 +L113¢23), 

m22 =!2+13+ m2r3 sg m3(L5 + a + 2L2r3¢3), 

M23 = M32 = 13 + m3(r3 +L31r3¢3), 

m33 =/3+ m3/5, 
and m; is the mass of link i, |; is the inertia of link i about its center of mass, and cp = cos 62,c3 = cos 63, 
c23 = cos(82 + 93). 

The 3 x 2 input matrix T(q) consists of two of the three columns [1, 0, 0}, [0, 1,0]7, and [0, 0,1], 
depending on which actuator is missing. This system can be written in the form (Equations 42.21 and 
42.22). Accordingly, this system is subject to an acceleration constraint. If the missing actuator is at 
the first joint, the acceleration constraint integrates to a velocity constraint—the angular momentum 


of the robot is conserved. The acceleration constraint does not integrate to a velocity constraint when 
the actuator is missing at the second or the third joint. 


Example 42.7: Underactuated Surface Vessel 


The underactuated surface vessel is another system with k = 0, but now with added damping. 
The problem is to control the Cartesian position and orientation of a vessel with two independent 
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FIGURE 42.5 Underactuated surface vessel. 


propellers as shown in Figure 42.5. The kinematic model describing the geometrical relationship 
between the earth-fixed (I-frame) and the vehicle-fixed (B-frame) motion is given as 


X = Vx Cos p — vy sin, (42.27) 
y= Vx sin + vy cos, (42.28) 
iy, (42.29) 


where (x,y) denotes the I-frame position of the center of mass of the vehicle, denotes the ori- 
entation, and (vx, vy) and wz are the linear and angular velocities of the vehicle in the B-frame. For 
simplicity, the origin of the B-frame is assumed to be located at the center of mass of the vehicle. It 
is also assumed that the vehicle is neutrally buoyant. Then the dynamic equations of motion of the 
vehicle can be expressed in the B-frame as 


Mv + C(v)v + D(v)v = 1, (42.30) 


where v = [vx, Vy, oz) denotes the velocity vector; t = [F,,0, T,]’ denotes the vector of external 
force and torque generated by the two propellers; M € R3*3 is the inertia matrix, including hydrody- 
namic added mass; and C(v) € R?*3 and D(v) € R3*3 denote the Coriolis/centrifugal and damping 
matrices, respectively. The following simplified model can be obtained by assuming that both the 
inertia matrix M and the damping matrix D are constant and diagonal: 


M11Vx — M22Vy Wz + d11Vx = Fx, (42.31) 
M2Vy +1141 Vx0z + daavy = 0, (42.32) 
M33 @z + (M22 — M11) VxVy + d330z = Tz, (42.33) 
where mj, dij, i= 1, 2,3, are positive constants. 


The surface vessel considered here has no side thruster, that is, Fy = 0, but the controllability 
analysis and control synthesis can be easily extended to the cases where t = [Fx. Fy, 01" or t= 
[0, Fy, Tz]’. 

Let (E, n) denote the position of the center of mass of the vehicle in the B-frame, which is given by 


(E,n) = (xcosw+y sin wW, —x sin +y cos Wp). 


Denote by 0 = S' x R? the configuration space parameterized by q=[w,&, nl’. 
By defining control input transformations from (Fx, Tz) to new control inputs (u;, U2) the equations 
of motion can be written as 


ea (42 34) 
Sig: (42.35) 
= uy — ol) + We) — (1+ p bE +B b7n, (42.36) 


where a = dz/mM2 and B = m41/m22. 
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Equation 42.36 can be rewritten as 
= 8b — al + bs) — (1+ BE + B72. 
This equation represents a nonintegrable relation involving not only the generalized coordinates 


and velocities but also the generalized accelerations and, hence, it can be viewed as a second-order 
nonholonomic constraint. 


42.3 Controllability 


In this section we briefly review some nonlinear controllability definitions. We then provide tests for 
these properties, including tests specific to mechanical systems. Further reading on definitions and tests 
for controllability can be found in [2,11,13,20,24]. 


42.3.1 Controllability Definitions 


Let V be a neighborhood of a point x € M. Let RY (x, T) indicate the set of reachable points at time T by 
trajectories remaining inside V and satisfying Equation 42.1, and let 


RiGee) = |) RE Ge 
0<t<T 


We define the following versions of nonlinear controllability: 


+ The system is controllable from x if, for any xp € M, there exists a T > 0 such that xp € RM a T). 
In other words, any goal state is reachable from x in finite time. 

¢ The system is accessible from x if RM (x, < T) contains a full-dimensional subset of M for some 
T>0. 

« The system is locally strongly accessible from x if RY (x, T) contains a full n-dimensional subset of 
M for all neighborhoods V and all sufficiently small T > 0. 

¢ The system is small-time locally accessible (STLA) from x if RY (x, < T) contains a full n-dimensional 
subset of M for all neighborhoods V and all T > 0. This is a weaker property than strong accessi- 
bility. 

¢ The system is small-time locally controllable (STLC) from x if RY (x, < T) contains a neighborhood 
of x for all neighborhoods V and all T > 0. 


The phrase “small-time” indicates that the property holds for any time T > 0, and “locally” indicates 
that the property holds for arbitrarily small (but full-dimensional) wiggle room around the initial 
state. 

Small-time local controllability is of particular interest. STLC implies that the system can locally 
maneuver in any direction, and if the system is STLC at all x € M, then the system can follow any 
curve on M arbitrarily closely. This allows the system to maneuver through cluttered spaces, since 
any motion of a system with no motion constraints can be approximated by a system that is STLC 
everywhere. 

For second-order mechanical systems, by definition STLC can only hold at zero velocity states. Nonzero 
velocity means that in small time, the system will necessarily have changed its configuration in the direc- 
tion of the velocity, effectively placing a unilateral constraint on the small-time reachable configurations. 
Also, for mechanical systems with velocity constraints, controllability and accessibility questions only 
make sense on the (2n — k)-dimensional distribution D C TQ. 

For mechanical systems, we are often particularly interested in understanding the reachable configu- 
rations irrespective of the velocity. This is a more restricted question than understanding the reachable 
states. Beginning from an equilibrium state x = [q',07]", a system is small-time locally configuration 
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accessible (STLCA) from q if the locally reachable set is full-dimensional on Q, and small-time locally 
configuration controllable (STLCC) from q if the locally reachable set on Q contains q in the interior. 
A stronger condition than STLCC is small-time local equilibrium controllability (STLEC) from q, which 
holds if the locally reachable set contains zero velocity states forming a neighborhood of q on Q. STLEC is 
stronger than STLCC, as STLEC demands that nearby configurations be reachable at zero velocity, while 
STLCC says nothing about the velocity. Finally, the system is equilibrium controllable if any equilibrium 
state can be reached from any other equilibrium state. 

In some cases, second-order mechanical systems can be modeled by first-order kinematic systems. 
Consider a second-order mechanical system written as 


k= f(x) + >> uigi(x), (42.37) 


i=l 
and a first-order kinematic system 


£ 


q= 2 wjvj(q), where q, vj(q) € R", (42.38) 
j=l 


where the velocity controls w; are continuous. The kinematic system (Equation 42.38) is a kinematic 
reduction of Equation 42.37 if all feasible trajectories of Equation 42.38 are feasible for Equation 42.37. 
We further say that a mechanical system (Equation 42.37) is maximally reducible to a kinematic system 
if there exists a kinematic reduction such that all feasible trajectories of the mechanical system, starting 
with an initial configuration qo and velocity in the column space of Y (qo) (defined in Equation 42.13), are 
also trajectories of the kinematic reduction. For example, all fully actuated mechanical systems are maxi- 
mally reducible to kinematic systems—we can equivalently assume the controls are forces or continuous 
velocities. 

If there exist kinematic reductions such that the kinematic model is STLC on configuration space, the 
second-order mechanical system is called small-time locally kinematically controllable (STLKC). STLKC 
implies STLEC. 


42.3.2 Controllability Tests 


42.3.2.1 General Controllability Concepts 


STLA and STLC are local concepts that can be established by looking at the Lie algebra of the system vector 
fields in a neighborhood of a state (i-e., using Taylor expansions of the vector fields at x). Accessibility 
and controllability, on the other hand, are global concepts. As a result, they may depend on things such 
as the topology of the space and nonlocal behavior of the system vector fields. In this section we focus on 
local tests. 
The Lie bracket of vector fields g; and g» is another vector field, given in local coordinates as 
[g1> 82] = 78 = 7, 

To order €”, this is the direction of the net motion obtained by flowing along the vector field g, for time ¢, 
& for time €, —g) for time €, and —g» for time €, where « < 1. We define the Lie algebra of a set of vector 
fields G, written Lie{G}, to be the linear span of all iterated Lie brackets of vector fields in G. For example, 
for G = {g1, 823}, the Lie algebra is spanned by gi, g2, g3, [g1,g2]5 [g1,93], [gi [gi> 92], [g3> [gi> [g2> 93], 
and so on. Each of these terms is called a Lie product, and the degree of a Lie product is the total number of 
original vector fields in the product. Lie products satisfy the Jacobi identity [g1, [g2, 31] + Lg2,[g3,g11] + 
[g3> [g1» 221] = 0, and this fact can be used to find a Philip Hall basis, a subset all possible Lie products that 
spans the Lie algebra [18]. 
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Theorem 42.1: 


The system (Equation 42.1) is STLA from x if (and only if for analytic vector fields) it satisfies the Lie algebra 
rank condition (LARC)—the Lie algebra of the vector fields, evaluated at x, is the tangent space at x, that 
is, Lie({f, g1,--->8m})(x) = TxM. If f =0, and by assumption the control set is UU = R", then the system 
is symmetric and the LARC also implies small-time local controllability. 


An early version of this result is due to W.-L. Chow, and it is sometimes called Chow’s theorem. 


Example 42.8: 


The control vector fields for the kinematic rolling disk are g; =[Rcos$,Rsind,1,0]’ and go = 
[0, 0, 0, 1]7. We define 


93 = [91,92] =[R sin, —Rcos $,0,0]", 
94 = [92,191,921] = [Ros o, Rsin ,0,0]", 


and note that G = {g1,...,g4} spans TyM at all x. Because f = 0, the rolling disk is STLC at all x. 


Second-order systems with nonzero drift are not symmetric for any control set. The system may still 
be STLC at zero velocity states, however, as symmetry plus the LARC is sufficient but not necessary for 
STLC. Sussmann [25] provided a more general sufficient condition for STLC that includes the symmetric 
case (f = 0) as a special case. To understand it, we first define a Lie product term to be a bad bracket if the 
drift term f appears an odd number of times in the product and each control vector field gj, i= 1,...,m, 
appears an even number of times (including zero). A good bracket is any Lie product that is not bad. For 
example, [g1, [f, g1]] is a bad bracket and [g2, [g1, [f, gi JJ] and Lg1; [g2, [g1,.g2]]] are good brackets. With 


these definitions, we can state a version of Sussmann’s theorem: 


Theorem 42.2: 


The system (Equation 42.1) is STLC at x if 


1. f(x) =0 

2. The LARC is satisfied by good Lie bracket terms up to degree k 

3. Any bad bracket of degree j < k can be expressed as a linear combination of good brackets of degree 
less than j 


The intuition behind the theorem is the following. Bad brackets are called bad because, after generating 
the net motion obtained by following the Lie bracket motion prescription, we find that the controls u; only 
appear in the net motion with even exponents, meaning that the vector field can only be followed in one 
direction. In this sense, a bad bracket is similar to a drift field, and we must be able to compensate for it. 
Since motions in Lie product directions of high degree are essentially “slower” than those in directions with 
a lower degree, we should only try to compensate for bad bracket motions by good bracket motions of lower 
degree. If a bad bracket of degree j can be expressed as a linear combination of good brackets of degree 
less than j, the good brackets are said to neutralize the bad bracket. For the bad bracket of degree 1 (the 
drift vector field f), there are no lower degree brackets that can be used to neutralize it; hence we require 
f(x) = 0. Therefore, this result only holds at states x where the drift vanishes, that is, equilibrium states. 
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Example 42.9: 


For the dynamic rolling disk, the configuration variables are [x1,X2,x3,x4]7 =[v, ox, yl", the 
drift vector field is f = [xs5,x6,X5R COS xX7,X5R Sinx>,0,0]’, and the control vector fields are n= 
[0,0,0,0,1/U-+ mR2),0]" and gz = [0,0,0,0,0,1/!]", each defined on the six-dimensional distri- 
bution D. We define the good Lie brackets g3 =[f,g1], ga =[f, ga], 95 =[Lf, 911, [f, g2]], 96 = 
(Lf, 911, {f, ga], (f, g2]], and observe that G = {g1,...,g6} spans TyM at all x. By Theorem 42.1, 
the system is STLA at all x. The maximum degree of good Lie brackets in G is six, and we find that all 
bad brackets of equal or lesser degree (degree 1, 3, and 5) are zero or can be neutralized by good 
brackets of lower degree. Therefore, at zero velocity states where f(x) = 0, the dynamic rolling disk 
is STLC. 


42.3.2.2 Simple Mechanical Systems with Zero Potential 


For the case of second-order mechanical systems (Equations 42.7 and 42.8) with no damping or potential 
terms (U(q) = 0), we can use the structure of the Lie brackets to derive simplified controllability tests. 
See [11,13] for details. 

We define the m columns of Y(q) (from Equation 42.13) as the input vector fields y;(q),..-,7m(q). 
The (constrained) covariant derivative of y2(q) with respect to y;(q) is 


Oy2(q) 


Vy (q)2(4) = Pig) ( 22, (q)+Ml(qQ)yp (tata). (42.39) 


With this definition, we can define the symmetric product of y; and y2 as the vector field 


(v1? ¥2) = Vy y2+ Vy. 


We can use the symmetric product in simplified controllability calculations for mechanical systems with 
no potential. The symmetric product captures patterns that appear when taking Lie brackets of the full 
system vector fields f(x) and g;(x). Advantages of using the symmetric product are that the vector fields 
have half the number of elements, and controllability properties may be established by products of lower 
degree than with Lie brackets. As a result, the number of controllability computations is reduced. 

Let Y = {y1(q),---+¥m(q)} denote the set of input vector fields, and let Sym(V) be the span of V and 
its iterated symmetric products. A symmetric product is “bad” if each of the vector fields appears an even 
number of times, and is “good” otherwise. 


Theorem 42.3: 


Beginning from an equilibrium state x =[q",0"]', the system (Equation 42.13) with zero potential is 


1. STLA from x if and only if Sym(V)(q) = Dg and Lie(D)(q) = Tz Q. 

2. STLC from x if Sym(Y)(q) = Dg and Lie(D)(q) = T,Q, and every bad symmetric product can be 
expressed as a linear combination of good symmetric products of lower degree. 

3. STLCA from q if and only if Lie(Sym(V))(q) = Tq. 

4. Both STLCC and STLEC from q if Lie(Sym(Y))(q) = T,Q and if every bad symmetric product can 
be expressed as a linear combination of good symmetric products of lower degree. If these conditions 
are satisfied at all q € Q, then the system is equilibrium controllable. 


We can also use the symmetric product to determine if the mechanical system is reducible to an 
equivalent kinematic system. 
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Theorem 42.4: 


A second-order mechanical system is maximally reducible to a kinematic system if and only if Sym(Y) = 


span(y). 


Even when the underactuated mechanical system is not fully reducible to a kinematic system, it may 
admit one or more kinematic reductions consisting of a single vector field. Such a kinematic reduction is 
also known as a decoupling vector field—a velocity vector field that the full mechanical system can follow 
at any speed. 


Theorem 42.5: 


A velocity vector field v(q) is a decoupling vector field of the second-order mechanical system if and only if 
v € span(Y) and Vyv € span(y). 


Example 42.10: 


The input vector fields for the dynamic rolling disk are computed to be y1(q) = UU + mR2)—! (0/dw + 
(R cos )0/Ox + (R sin )0/dy) and y2(q) = (1/1)0/0. The disk is trivially shown to be STLC at zero 
velocity states by Theorem 42.3 as span({y1,¥2})(q) = Dg for all gq, and we have already shown 
that Lie(D)(q) = TgQ. Further, by Theorem 42.4, the disk is maximally reducible to the kinematic 
disk system, with velocity vector fields v; = y1, v2 = y2, and the system is STLKC with these vector 
fields. 


Example 42.11: 


For the snakeboard example, the two input vector fields y;, y2 of Y(q) in Equation 42.13 correspond 
to torque rotating the rotor and the wheels, respectively. The snakeboard is STLC at zero velocity 
states, and therefore STLCC and STLEC, by Theorem 42.3. The vector fields y;, yz are also decoupling 
vector fields making the system STLKC. The snakeboard is not maximally reducible to a kinematic 
system, however. 


Example 42.12: 


For the 3R robot with a missing actuator at the first joint, the acceleration constraint integrates 
to a velocity constraint (conservation of angular momentum), and therefore the system is con- 
strained to a lower-dimensional subspace of TQ. This system is maximally reducible to a kinematic 
system. 

Fora missing actuator at the third joint, the system admits two decoupling vector fields: translation 
along the length of the third link and rotation of the third link about a point on the link a distance 
r3 +13/m3r3 from the third joint. (This point is known as the center of percussion of the link with 
respect to the third joint.) The 3R robot is STLC at zero velocity states, STLCC, STLEC, and STLKC away 
from the singularity where 92 € {0, 1}. It is not maximally reducible to a kinematic system, however. 
See [12] for details. 
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42.3.2.3 Mechanical Systems with Nonintegrable Acceleration Relations 
and No Velocity Constraints 


This section develops controllability and stabilizability results for underactuated systems with completely 
nonintegrable acceleration relations. 

Consider Equations 42.21 and 42.22, and let I, denote the set {1,...,p} for any integer p > 1. Define 
the n — m covector fields 


w! = > Jii(qaqu,j _ dq2,i + Ri(q, q)dt, i€In_-m (42.40) 
j=l 


on M x R so that the n—m relations given by Equation 42.22 can be rewritten as w! = 0, i € In—m. 
Augment the covector fields (Equation 42.40) with 


oy = dqijj _ qi jat, je Im (42.41) 


a™* = day. —Gopdt, k €In—m (42.42) 


and let Q Cc T*(M x R) denote the codistribution 
Q = span{w', , i € In—m, j € In}. (42.43) 


The annihilator of 2, denoted 2+, is spanned by m + 1 linearly independent smooth vector fields 


a 
ae zm a Z + Rx, 5p) + Bi (42.44) 


ee Wild 5 GEL (42.45) 


We present the following definition. 


Definition 42.1: [21] 


Consider the distribution Q+ and let C denote its accessibility algebra, that is, the smallest subalgebra 
of V°(M x R) that contains to, t1,..., Tm. Let C denote the accessibility distribution generated by the 
accessibility algebra C. Then the acceleration relations defined by Equation 42.22 are said to be completely 
nonintegrable if 

dim C(x,t)=2n+1, V(x,theMxR. 


Note that the above definition gives a coordinate-free characterization of nonintegrability for any set 
of acceleration relations of the form Equation 42.22. Note also that this definition is analogous to the 
definition given in [2] for the nonintegrability of a set of kinematic or velocity relations. 

Examples of underactuated systems with completely nonintegrable acceleration relations include 
underactuated robot manipulators (such as our 3R robot), underactuated marine vehicles, the planar 
vertical takeoff and landing (PVTOL) aircraft, the rotational translational actuator (RTAC) system, the 
acrobot system, the pendubot system, and examples in [21]. 

In the rest of this section, we assume that the acceleration constraints (Equation 42.22) are completely 
nonintegrable. With this assumption, the underactuated mechanical system (Equations 42.21 and 42.22) 
is strongly accessible [21]. 
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A particularly important class of solutions of Equations 42.21 and 42.22 are the equilibrium solutions 
with v(t) = 0, Vt > 0. A solution is an equilibrium solution if it is a constant solution. If (q, q) = (qe, 0) 
is an equilibrium solution, we refer to qe as an equilibrium configuration. Clearly, the set of equilibrium 
configurations of the system (Equations 42.21 and 42.22) is given by 


{q € Q| R(q, 0) = 0}. 


It is well-known that strong accessibility is far from being sufficient for the existence of a feedback 
control that asymptotically stabilizes the underactuated system at an equilibrium solution. In certain cases 
it is possible to prove STLC at equilibrium states, which guarantees the existence of a piecewise analytic 
feedback law for asymptotic stabilization in the real analytic case. Since an underactuated mechanical 
system satisfies 1 <m <n, the dimension of the state is at least four. Hence, in the real-analytic case, 
the STLC property also guarantees the existence of asymptotically stabilizing continuous time-periodic 
feedback laws (see, for example, [21] and the references therein). 

We now briefly summarize a result of Bianchini and Stefani [1] that generalizes Theorem 42.2 and is 
utilized to prove the subsequent controllability results. Let Br(X) denote the smallest Lie algebra of vector 
fields containing f, g1,...,2m and let B denote any bracket in Br(X). Let 8°(B), 8!(B),...,8”"(B) denote 
the number of times f, 1,..., 8m, respectively, occur in the bracket B. For an admissible weight vector 
1= (Ip, 4,...51m), 1) = Io = 0, Vi, the l-degree of B is equal to the value of )7/" 5 1;8'(B). The Bianchini and 
Stefani condition for STLC for a strongly accessible system is essentially that the so-called bad brackets, 
the brackets with 5°(B) odd and 8/(B) even for each i, must be I-neutralized, that is, must be a linear 
combination of good brackets of lower I-degree at the equilibrium. 

Consider the system Equations 42.21 and 42.22, and rewrite the drift and control vector fields as 


f=141.42,0°. Ral", 
3 =(07,07 67 JQ)", je In. 
The following Lie bracket calculations are straightforward: 
[gi gil=0, if € Ins 
[fg] =[-e!, Ji (q),07 #)", i Ins 
[gi [f-gil] =[07,07,07, Hi (Q)]", if €Ims 
Uf [gj Uf gill] = (07, Hy (q),07,*]", 67 € Im; 


where 
Ola), Ij(q) a (Mae ' ) ie 
Aiyj(q) = 3g hj(q) + 4 hi(q) ai 3 hj(q) ) hq), i.7 € Im (42.46) 
ej ; 
h(q) = EA » Len. (42.47) 


Note that the vertical lift of h; (considered as a vector field on the configuration space Q) is the control 
vector field g;. Note also that Hyj(q) = Hyi(q), Vq € Q, Vi, j € Im. 
We now present the following result. 


Theorem 42.6: 


Letn—m > 1 and let (qe, 0) denote an equilibrium solution. The underactuated mechanical system, defined 
by Equations 42.21 and 42.22, is STLC at (qe, 0) if there exists a set of n—m pairs of indices (ig, jp) € i 
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such that 
dim span{Hi,j, (qe), ik # jks k €In-m} =n—m (42.48) 


and Hi;(qe), i € Im, can be written as a linear combination of Hixj,(Ge)> ix A jk» k € In—m, such that Ip + 
21; > lo +h, +1, for some admissible weight vector 1 = (Ip, h,.--lm), li = Io = 0, Vi. 


Note that for the condition (Equation 42.48) to hold, the condition m(m-+ 1) > 2n must be satisfied. 
This condition arises due to the fact that in the above result we have considered Lie brackets up to degree 
four only. It is possible to develop a result that weakens or even removes this restriction by also taking 
into account higher-order Lie brackets. 

We now restrict our consideration to underactuated mechanical systems with no potential or friction 
forces. For such systems, when evaluated at the equilibrium, the only nontrivial brackets are those satisfy- 
ing )-, 8'(B) — 8°(B) = 0 or 7°, 8/(B) — 8°(B) = 1. Clearly, the brackets with }-” , 8/(B) — 8°(B) = 0 
are all good, and the only bad brackets are those with }-/", 8/(B) — 8°(B) = 1, 8°(B) odd, and 8‘(B) even, 
Vie Im. 

We define the following sequence of collections of vector fields: 


G, = {gi, i€ Im}, 
Ge = {IX [f, YI, X€Gi, YEG, k=it+j}, k>=2, 


G=|JGi. 


i>2 


Let X denote a vector field in G. It is easy to show that X has the form X = [o?,o7, 0%, N'(q)] T where N(q) 
is an n — m vector function, and its Lie bracket with f can be written as [f, X] = [o!, —N7™(q), Of xq]. 
Now let (qe, 0) denote an equilibrium solution. Clearly, if there exists an integer k* > 2 such that 


k* 
dim span 4 X(qe,0), X € U Git =n—™m, (42.49) 
i=2 


then the system is strongly accessible at (qe, 0), that is, the system satisfies a necessary condition for STLC 
at (qe, 0). As shown in [21], all the bad brackets can be written as linear combinations of the bad brackets 
contained in G. Thus, a sufficient condition for STLC at (qe, 0) can be obtained by considering the bad 
brackets in G and applying the Bianchini and Stefani condition. 

The following result can now be stated. 


Theorem 42.7: 


Letn —m > land let (qe, 0) denote an equilibrium solution. Consider the underactuated mechanical system, 
defined by Equations 42.21 and 42.22, and assume that the components of R(q, q) are of second-order in 
q-variables. Also assume that the condition (Equation 42.49) is satisfied. Then, the system is STLC at (qe, 0) 
if there exists an admissible weight vector 1 = (lp, h,...51m)s 1 = Io = 0, Vi, such that every bad bracket in 
Gaz, k € Zt, can be |-neutralized. 


Example 42.13: 


Consider the surface vessel dynamics described by Equations 42.34 through 42.36. It can be shown 
that the following hold: 
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1. The system is strongly accessible since the space spanned by the vectors 


91, 92, [f, 91], [f, 92], (92, lf, [f, (g2, lf. 9111) 


has dimension 6 at any (q,q)<€ M=TQ. 

2. The system is STLC at any equilibrium (qe, 0) since the sufficient conditions for STLC of Theo- 
rem 42.6 are satisfied with the admissible weight vector I = (2, 4, 3). 

3. There exist both time-invariant piecewise analytic feedback laws and time-periodic continuous 
feedback laws that asymptotically stabilize (qe, 0). 


These controllability properties guarantee the existence of the solution to the problem of control- 
ling the surface vessel to any desired equilibrium configuration. The time-invariant discontinuous 
feedback control laws developed in [22] represent such solutions. 


42.4 Feedback Stabilization 


A beautiful general theorem on necessary conditions for feedback stabilization of nonlinear systems was 
given in [9]. 


Theorem 42.8: Brockett 


Consider the nonlinear system x = f(x,u) with f(xo,0) =0 and f(-,-) continuously differentiable in a 
neighborhood of (xo, 0). Necessary conditions for the existence of a continuously differentiable control law 
for asymptotically stabilizing (xo, 0) are: 


1. The linearized system has no uncontrollable modes associated with eigenvalues with positive real 
part. 

2. There exists a neighborhood N of (xo, 0) such that for each & € N there exists a control u(t) 
defined for all t > 0 that drives the solution of x = f(x, us) from the point x =§& at t=0 to 
x=xp att=oo. 

3. The mapping y: N x R™ — R", N a neighborhood of the origin, defined by y : (x, u) > f(x, u) 
should be onto an open set of the origin. 


Based on an extension of this result to continuous feedback, the key consequence for kinematic systems 
is the following. 


Proposition 42.1: 


Consider the system 


m 
k= \o ugk, (42.50) 
k=1 


where x € R", m <n, and 
rank{g;(0),...,2n(0)} =m. 
There exists no continuous feedback that locally asymptotically stabilizes the origin. 


One way to deal with this problem is to use time-varying feedback. A key result on stabilization by 
time-varying feedback is that of [15]. Another approach is nonsmooth feedback (see below). 
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42.4.1 Kinematic Example: The Heisenberg System 


In this section we consider the problem of feedback stabilization of the Heisenberg system Equations 42.4 
through 42.6 to an equilibrium or a trajectory. This serves as a model system for the kinematic vertical disk 
and many wheeled mobile robots, for example. We then extend the results to the generalized Heisenberg 
system, which can be shown to be equivalent to a broad class of first-order nonholonomic systems. 


42.4.1.1 Stabilization of the Heisenberg System 


One of the possibilities suggested in [3] is to utilize sliding modes to stabilize the Heisenberg system, by 
applying the following feedback control law: 


uy = —ax, + x2 sign(z), (42.51) 
uz = —Ax2 — Bx sign(z), (42.52) 


where « and 6 are positive constants. It is shown that if an initial condition is outside of the parabolic 
region defined by the inequality 
B 


ria + x3) < |z\, (42.53) 


then the control (Equations 42.51 and 42.52) stabilizes the state z to zero in finite time, and on the 
manifold z = 0 the two remaining states x, and x2 converge to the origin too. If the initial data are such 
that Equation 42.53 is true, that is, the state is inside the paraboloid, we can use any control law which 
steers it outside, and then apply (Equations 42.51 and 42.52). 

Let us note here that because of the switchings, the above strategy assumes that the control input is 
a discontinuous function of the state variables. In such a system the sliding mode may exist when the 
solution moves along the set of discontinuity of the right-hand side (as happens with Equations 42.51 and 
42.52 when z = 0). Thus, the existence of the corresponding solution of the differential equations (off the 
z-axis only) should be understood in the sense of the Filippov definition. 

A similar way to stabilize system (Equations 42.4 through 42.6) to the origin is the following variable 
structure control law: 


uy = —Ax, + Bzx2, (42.54) 
U2 = —AX2 — Bzx1, (42.55) 


where & = a(x), X2,Z) and B = B(x1, x2, Z) are positive state-dependent switching functions. 
With this choice, the system (Equations 42.4 through 42.6) becomes 


x, = —ax, + Bzx2, (42.56) 
X2 = —AX2 — Bzx1, (42.57) 
z= —B2(x? + x3). (42.58) 


There are a number of strategies for choosing a and f to stabilize the system. Let V = x7 +x3, then 
V = —2aV. It is clear from Equation 42.58 that if we initially choose a = 0 and B > 0, then for x or x2 
not equal to zero, z will be driven asymptotically to 0, while V will remain fixed. On the other hand, for 
a > 0 and f =0, V will be driven to 0. Thus, the stabilizing control in this case should be the following. 
(1) unless V = 0, apply a = 0 and B > 0. This guarantees z — 0. Then (2) apply a > 0 and 6 = 0. In this 
case, while z remains zero, V — 0. If initially V = 0, apply any other nonzero control to make V £0, 
then apply (1). The convergence in phase (1) is asymptotic, but it is clear that we can achieve stabilization 
with any desired accuracy by alternating steps (1) and (2), if necessary. 

As we show below, this strategy can be extended to the generalized system (Equations 42.80 and 42.81). 
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42.4.1.2 Tracking in the Heisenberg System 


The robustness property of the sliding mode algorithm (Equations 42.51 and 42.52) allows trajectory 
tracking for arbitrary smooth three-dimensional trajectories, even though the control dimension is equal 
to two. 

Let X*(t) = [xf (4), x3 (0), z*(t)]" be an arbitrary smooth curve in R>. Let Z be defined as 


Z(t) = z(t) — x} (t)xa(t) + x5 (t)x1 (2). (42.59) 


Using Equations 42.4 through 42.6 and 42.59, the derivative of Z can be written as 


2 = (xy — x*)(ug — x2") — (x2 — x3)(ur — 4) +. g(t, x10), (42.60) 


where 


g(t, X1,X2) = Ixy x5 — > xf — Wxokf + KF x5. (42.61) 


The problem of tracking x}, x3, z* by the variables x1, x2, z is equivalent to the one of stabilizing 
Xj =X] xf, X2 = X2 x, Z=2 Zo 


The system (Equations 42.4 through 42.6) in the new variables can be written as 


x1 =i, (42.62) 
X. = i, (42.63) 
Z =X iy — Xi +B, (42.64) 


where we used the notations # = u) — xf, U2 = un -—*x5,g=g-—2". 


If g = 0, that is, the desired trajectory satisfies the constraint 


2 = xfK — x5xT, (42.65) 


then the system (Equations 42.62 through 42.64) corresponds exactly to the Heisenberg system, and we 
can use the control Equations 42.51 and 42.52 or Equations 42.54 and 42.55 described in the previous 
section. 

If g £0, then unlike in the previous case, perfect tracking is not possible, but we can track with 
arbitrary accuracy. In this case we can still apply control of the type (Equations 42.51 and 42.52), but with 
state-dependent a and B: 


ay = —aX) + BX? sign(Z), (42.66) 
ig = —aX2 — PX, sign(Z). (42.67) 


Substituting into Equations 42.62 through 42.64, we obtain 


x, = —aX, + Px, sign(Z), (42.68) 
Xp = —AX, — Bx sign(Z), (42.69) 
z= —B(x? + X3) sign(Z) +g. (42.70) 


Here we assume that « is the following function of x; and x2: 


LY ate id) 2 
as {*° a tee (42.71) 


oy if XP +X <e, 
where do > 0 and a; < 0 are constants, and ¢ > 0 is a constant that defines the tracking accuracy. Let us 


consider the Lyapunov function V = x7 + x3. As in the previous case, its derivative along the trajectories 
of the system (Equations 42.68 through 42.70) is V =—2aV. Due to this choice of a, the function 
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V converges to V =e” =const. for any initial conditions except for the origin. This means that for 
sufficiently large B such that 


Be* > |g| (42.72) 
and for sufficiently large t in Equation 42.70, 
z= —fV sign(z) +2 (42.73) 
we obtain sliding mode on the manifold 
z=0 fort>h, (42.74) 


where the time t; can be chosen arbitrarily close to the initial moment by increasing f. In general, B 
should be chosen to be a function of x1, x2, and X*, X* to satisfy (Equation 42.72), but due to the separate 
convergence of x), X2 to the e-neighborhood of the origin. For a bounded reference curve with bounded 
derivative B can be constant. 

Thus, the state trajectory X(t) = [x1 (t), x2(t), z(t)] enters an e-neighborhood of the curve X*(t) not 
later than in time f) and stays in that neighborhood for all subsequent moments of time. The generalization 
of the e-solution of the tracking problem for the case of higher-order systems can be found in [16]. 


42.4.1.3 The General Setting 


In Section 42.2.1, we used the rolling vertical disk example to demonstrate the equivalence of the Heisen- 
berg system to a particular class of nonholonomic systems. The generalization of this equivalence, at least 
locally, was obtained by Brockett [8]. He showed that controllable systems of the form 


x = B(x)u, (42.75) 


where u € R” and x € R""+1)/? is such that the first derived algebra of control vector fields spans the 
tangent space TR™"*+))/? at any point, is locally equivalent to 


=U, (42.76) 
Y=xu' — ux’, (42.77) 


where x and u are column vectors in R” and Y € so(n), n > 2. Here so(n) is the Lie algebra of n x n skew 
symmetric matrices: Y' = —Y. 

The system (Equations 42.4 through 42.6) is a particular case of the so(n) system (Equations 42.76 and 
42.77). That can be easily seen if we identify the variable z with the skew-symmetric matrix 


and observe that 


x) a 0 xX, U2 — X2U, 
[u1, u2] — [x1, x2] = . 
x2 u2 —x1 U2 + X2U 0 


A different generalization of the Heisenberg system (Equations 42.4 through 42.6) is obtained by 
identifying the vectors [x), x9]? and [u1, v2]! with the matrices 


1 xX) —X2 1 uy a 
xX =— and U=-—— 
J2 (ie | J/2 Ee uy 
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respectively. Then we have 


[U, xX] = ux-xu=| i paean 
—xX1U2 + x2Uy 0 
This suggests the following matrix system occurring in the Lie algebra s/(n,R) of n x n matrices with 


trace 0: 


X=U, (42.78) 
Y =[U, X], (42.79) 


where X, U € symo(n, R) and Y € so(n). Here symo(n, R) is the space of n x n real symmetric matrices 
with zero trace. Note that s/(n, R) = symo(n, R) @ so(n). 

The system that generalizes both the so(n) system (Equations 42.76 and 42.77) and the sI(n, R) system 
(Equations 42.78 and 42.79) is the following system on a Lie algebra: 


k= u, (42.80) 
Y =[u,x], (42.81) 


where x,u €m, Y € h such that h is a Lie subalgebra of a real semisimple Lie algebra g, and m is the 
orthogonal complement of relative to the Killing form B: g x g > R. The algebra g has the structure of 
a direct sum decomposition g = mb, [h, m] C m, and [m, m] = h. We note that every simple Lie algebra 
with a Cartan decomposition is of this type. 

The problem we consider herein is that of finding a stabilizing control for the general system (Equations 
42.80 and 42.81). Since the dimension of m, which is where the control u takes its values, is smaller than 
the dimension of the state space g, the system fails Brockett’s necessary condition for the existence of a 
continuous feedback law (see [9]). A discontinuous feedback for the state stabilization of the generalized 
Heizenberg system can be found in [3,4]. It completely solves the stabilization problem for Equations 42.80 
and 42.81. 

We now consider the general setting for stabilization of the system (Equations 42.80 and 42.81) in 
g =m b. The main part of the variable structure control is given by 


u=—ax+ 6[Y,x]—yLY,[Y,x]] =—ax+BlY,x]+yN(Y)x, (42.82) 


where a, B, y : g > R are switching functions, and N(Y) = —(ad Y)*|m isa nonnegative symmetric oper- 
ator on m relative to the inner product. We will also need M(x) = «(ad x)*|y, a nonnegative symmetric 
operator on h relative to the inner product. Here 


(42.83) 


oe 1, if gis of noncompact type 
~ |-1, if gis of compact type 


Note that (ad x)? is nonnegative if g is noncompact, and nonpositive if g is compact, and therefore, 
introducing € into the definition of M(x) guarantees that it is a nonnegative operator. We will assume 
that a, y > 0 and Be < 0. 

With (Equation 42.82) as our choice of u, the system (Equations 42.80 and 42.81) becomes 


x=-—ax+B[Y,x] + yN(Y)x, (42.84) 
Y =BeM(x)¥ + y[N(Y)x, x] = BeM(x)Y — ye[Y, M(x)Y]. (42.85) 


Using (Equations 42.84 and 42.85), we compute 


d 
qe = —2a|)x||? + 2y(x, N(Y)x) (42.86) 


Control of Nonholonomic and Underactuated Systems 42-25 


and 


“| Y||? = 2Be(Y, M(x)Y). (42.87) 


Since Be < 0 and M(x) is a nonnegative operator, it follows that the right-hand side of Equation 42.87 is 
nonpositive. Thus, || Y|| is nonincreasing in general, and is constant if B = 0. 

Our stabilization algorithm will be (necessarily) discontinuous, and will require switching of the control 
(Equation 42.82) between the following three cases 1-3 and case 4: 


l. a>0,B=y=0 

2. @=Kh,, y= kK, and B = 0, where X,, is the largest eigenvalue of N(Y) 

3. a=y=0, pe < 0 

4. u=—a(x —z,), where a > 0, and z, denotes a fixed ,.-eigenvector of N(Y) 


In case 1, x is driven to 0 radially while Y remains fixed. If Y was not already 0 in the first place, imple- 
menting the control (Equation 42.82) with these parameter values will render the system unstabilizable. 
Thus, this case will only be used if Y = 0. 

In case 2, ||Y|| is a constant and the spectrum of ad Y is constant. Therefore, the spectrum of the 
operator N(Y) is constant, as are the dimensions of its eigenspaces. Let \,, denote those eigenvalues of 
N(Y) in this case, we have that Y evolves isospectrally with constant norm and asymptotically vanishing 
velocity, while x is driven to x,, its (constant) projection onto the i,.-eigenspace of N(Y). 

In case 3, ||x|| is a constant. If we let Y; denote the projection of Y onto the nullspace of M(x), then 
noting that Ys = Y+|;=0 is constant, we conclude that Y — Ys asymptotically. Summarizing, in this case, 
we have that x evolves isospectrally with a constant norm, Y is driven to Yz, its (constant) projection onto 
the nullspace of M(x), and using the orthogonality of the eigenspaces, we can compute 


Yell? < YI? — Ax. (42.88) 


Case 4 is used to move x from the origin along the eigenvector corresponding to the biggest eigenvalue 
he of N(Y). 

The idea of the feedback strategy is the following. Since for Y = 0 step (1) stabilizes the system, the 
main task of the algorithm is to move Y to the origin. As mentioned above, step (2) allows to decrease 
the magnitude of || Y||? by X,, the biggest eigenvalue of N(Y). On the other hand, applying (2) guarantees 
that || Y|| and the spectrum of the operator N(Y) remain constant. Thus, Y evolves isospectrally with a 
constant norm, while x converges to the constant x,. If x, 4 0, then go to (3). Otherwise, use (4). Then x 
converges to z, while Y remains constant. 

As shown in [4], by alternating steps (1)-(4) the system can be stabilized in a finite number of steps. 
The resulting algorithm is a variable structure feedback. 


42.4.2 Energy Methods for Nonholonomic Mechanical Systems 
with Symmetries 


We discuss here stabilization of nonholonomic systems with symmetry. In the simplest setting, the 
configuration space Q is the direct product S x G, where S is a smooth manifold and G is the symmetry 
group.* The Lagrangian and the constraints are invariant with respect to the action of G by left translations 
on the second factor of the decomposition of Q. If the group G is Abelian, the group variables become 
cyclic. We assume here that the Lagrangian L : Q — R equals kinetic minus potential energy of the system, 
and that the kinetic energy is given by a quadratic form on the configuration space. 

In the presence of symmetry, we write the configuration coordinates as q = (1, g), where r € S is the 
shape variable and g € G is the group variable. The state coordinates are x = (1, g, 7, Q), where Q € g is the 


* In general, this direct product structure is observed only locally, while globally one sees a (principal) fiber bundle. 
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constrained group velocity relative to the so-called body frame (see [2] for details).* With these notations, 
Equations 42.11 are usually written as 


# = F(r,7,Q), (42.89) 
Q=FTA NLL QM (NQ+FH L(V, (42.90) 
fa e8 (42.91) 


as discussed in [2]. Note that controls have not been introduced yet. Equations 42.89 and 42.90 are called 
the shape and momentum equations, respectively. Equation 42.91 is called the reconstruction equation. 
Below, stability is understood in the orbital sense, and thus one can study system (Equations 42.89 and 
42.90). 


42.4.2.1 The Energy-Momentum Method 


Recall that relative equilibria are solutions of the full system (Equations 42.89 through 42.91) such that 
the shape and momentum variables are kept constant. In other words, after reduction relative equilibria 
of Equations 42.89 through 42.91 become equilibria of the reduced system (Equations 42.89 and 42.90). 
Stability of these equilibria implies orbital stability of corresponding relative equilibria. 

The energy momentum method is a stability analysis technique that uses the restriction of energy on 
the momentum levels as a Lyapunov function. The momentum is always conserved in holonomic systems 
with symmetry. 

Unlike the holonomic case, the momentum equation in the nonholonomic setting generically does not 
define conservation laws. Examples such as the rattleback and Chaplygin sleigh are well known. Some- 
times, however, the components of momentum relative to an appropriately selected moving frame are 
conserved. This is observed in examples like the balanced Chaplygin sleigh, where the angular momen- 
tum relative to the vertical line through the contact point of the body and the supporting plane and the 
projection of the linear momentum onto the blade direction are conserved. We remark that this kind of 
momentum conservation is significantly different from that of in holonomic systems, in which, according 
to Noether’s theorem, the spatial momentum is conserved. 

In the special case Y(r) = 0, I'(r) = 0, the momentum equation can be rewritten as 


dQ = dr™ A(r)Q. (42.92) 


Theorem 42.9: [26] 


If the distribution (Equation 42.92) is integrable, the system has conservation laws 
Q=Fir,c), (42.93) 
where c are constants. 


These conservation laws enable one to extend the energy-momentum method for stability analysis 
to nonholonomic setting. Let E(r,7,Q) be the energy of the system; since the system is G-invariant, 
the energy is independent of the group variable g. The energy itself is often not positive-definite at an 
equilibrium (re, Qe) of the reduced system, and thus the energy cannot be used as a Lyapunov function. If, 
however, the conditions of Theorem 42.9 are satisfied, one can construct a family of Lyapunov functions, 
one for each level (Equation 42.93). 


* 


Here and below, g denotes the Lie algebra of the group G. 
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Theorem 42.10: [2] 


Assume that the energy restricted to the level of the conservation law (Equation 42.93) through the equilib- 
rium (Te; Qe) of Equations 42.89 and 42.90 is positive-definite. Then this equilibrium is Lyapunov stable, 
and the corresponding relative equilibrium is orbitally stable. 


The statement of this theorem can be extended to a more general class of systems and used for 
establishing partial asymptotic stability of relative equilibria of nonholonomic systems. Details may be 
found in [2]. 


42.4.2.2 Energy-Based Feedback Stabilization 


Following [6], we now apply the energy-momentum approach to the problem of stabilization of 
relative equilibria of nonholonomic systems with symmetry. Recall that the dynamics is given by 
Equations 42.89 through 42.91. We assume that control inputs are G-invariant, are applied in the sym- 
metry directions, and are consistent with constraints. Thus, the controlled dynamics are G-invariant and 
the corresponding reduced controlled dynamics are given by 


* = F(r,7, Q), 
Q=7 ANQ4+Q™T(NQ+7 Vi + T(r)u, 


where u are the controls. Our strategy is to assign feedback control inputs that have the same structure 
as the right-hand side of the momentum equation. That is, the control inputs are given by homogeneous 
quadratic polynomials in + and {2 whose coefficients are functions of r. The controls are selected in 
such a way that the controlled momentum equation satisfies the conditions of Theorem 42.9, and thus 
the controlled momentum equation defines controlled conservation laws. We then utilize the remaining 
freedom in the control selection and make the equilibria of the system’s dynamics, reduced to the levels 
of the controlled conservation laws, stable. In other words, the controls are used to shape the momentum 
levels in such a way that the energy reduced to the level through the equilibrium of interest becomes 
positive-definite, thus letting us use the energy-momentum method to conclude stability. 

The proposed strategy is only capable of nonasymptotic stabilization. If partial asymptotic stabilization 
is desirable, one should add dissipation-emulating terms to the control inputs. In this case, stability is 
checked by the energy-momentum method, although it may be necessary to use methods for stability 
analysis for nonconservative systems such as the Lyapunov—Malkin theorem. See [2] for details. 

To simplify the exposition, the details are given for systems with one shape degree of freedom. 

Consider a system with Lagrangian 


L(r,7, 2) = K(r,7, 2) — U(r) 
and assume that the momentum equation of the controlled system is 
Q = (A(r)+ A(r))Q i. 
According to our general strategy, the distribution defined by this equation; 
dQ = (A(r) + A(r))Q dr 
is integrable and defines controlled conservation laws 


Q=F(r,c). (42.94) 


42-28 Control System Advanced Methods 


Dynamics on the levels of these conservation laws reads 
* = F(r,7,F“(r, ¢)). (42.95) 


This defines a family of one degree of freedom Lagrangian (or Hamiltonian) systems. The orbital 
stability analysis of relative equilibria of the original system reduces to the stability analysis of equilibria 
of Equation 42.95. This analysis is somewhat simpler to carry out if the Euler-Lagrange form of the 
equations of motion is used instead of Equation 42.95. 

Skipping some technical details, the Euler-Lagrange form of the dynamics on the levels of controlled 
conservation laws (Equation 42.94) is 


are é OU _ 
en?) + AF(r,6)) + 5 =0, (42.96) 


where Q(-) is a quadratic form. Therefore, the equilibrium (re, Qe) is stable if the following condition is 


satisfied: ae 
< (a Ge) * =) 


where Ce is defined by the condition 


> 0, (42.97) 


T=Pe 


Fre, Ce) = Qe. 


This stability condition is obtained by using the energy of dynamics (Equation 42.96) as a Lyapunov 
function, which is justified by the energy-momentum approach to stability analysis; see [2] for details. 

We illustrate the above techniques by the problem of stabilization of slow motions of a falling rolling 
disk along a straight line. 


42.4.2.3 Stabilization of a Falling Disk 


Consider a uniform disk rolling without sliding on a horizontal plane. The disk can reach any configura- 
tion, therefore the constraints imposed on the disk are nonholonomic. It is well-known that some of the 
steady-state motions are the uniform motions of the disk along a straight line. Such motions are unstable 
if the angular velocity of the disk is small. Stability is observed if the angular velocity of the disk exceeds a 
certain critical value; see [2] and [19] for details. Below we use a steering torque for stabilization of slow 
unstable motions of the disk. 

We assume that the disk has a unit mass and a unit radius. The moments of inertia of the disk relative 
to its diameter and to the line orthogonal to the disk and through its center are I and J, respectively. The 
configuration coordinates for the disk are [6, ts, @, x, y]’ as in Figure 42.6. Following [19], we select e; to 


*h 


FIGURE 42.6 The geometry and configuration coordinates for the falling rolling disk. 
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be the vector in the xy-plane and tangent to the rim of the disk, e2 to be the vector from the contact point 
to the center of the disk, and e3 to be e; x e2, as shown in Figure 42.6. 

At each q € Q, the fields e;, e2, and es span the subspace Dj defined by the constraint distribution D, 
the fields e and e3 span the constrained symmetry directions, and the dual of e2 is the control direction. 
The component of disk’s angular velocity along e; equals 6; the ey and e3 components are denoted by € 
and ». 

Using this frame, equations of motion are computed to be 


(I+ 1)6+ Ié” tand — (J + En —g sind =0, (42.98) 
1& — 16 tan + Jy = u, (42.99) 
J +1) +86=0, (42.100) 


where u is the steering torque and g is the acceleration of gravity. In the absence of the torque, the last 
two equations can be written as conservation laws of the form 


&=F¢5(0,ce,Cy), N= Fy (0, ce, Cy); (42.101) 


here and below the parameters c; and cy label the levels of these conservation laws. These conservation 
laws are obtained by integrating the equations 


de 7 dy 
IT, = [stan 0 Jn, ag = g, 


Now consider a steady-state motion 0 = 0, € = 0, n = ne. This motion is unstable if 7¢ is small. Set 
u=—f(0)n, (42.102) 


where f (9) is a differentiable function. The motivation for the choice (Equation 42.102) for u is that it 
preserves the structure of Equations 42.99 and 42.100, and thus the controlled system will have conser- 
vation laws whose structure is similar to that of the uncontrolled system. Viewing 9 as an independent 
variable, we replace Equations 42.99 and 42.100 with the linear system 


dg _ dn 
Io, = is tané (J +f(8))n, Oar Dag = g. (42.103) 


The general solution of system (Equation 42.103), 
E=Fe(0,c,e), n=F,(0,ce,cy), (42.104) 


is interpreted as the controlled conservation laws. The functions that define these conservation laws are 
typically difficult or impossible to find explicitly. 
Dynamics (Equation 42.96) on the level set of the conservation laws for the disk becomes 


(I+ 16+ Itan 6 (FE (0, ce, Cn)? — J + DFE, ce, Cy) FX (0, cz, Cn) — g sin ® = 0. 


The condition for stability of the relative equilibrium 0 = 0, & = 0, 7 = Ne is obtained using formula 
(Equation 42.97). Using Equation 42.103, the stability condition becomes 
Ig 
(0) > —-—, - J. (42.105) 
NS Oe Ine 
That is, any function f (8) whose value at 0 = 0 satisfies inequality (Equation 42.105) defines a stabilizing 
steering torque. 

Let us reiterate that in the settings considered here the energy-momentum method gives conditions 
for nonlinear Lyapunov (nonasymptotic) stability. Hence stabilization by the torque (Equation 42.102) 
is nonlinear and nonasymptotic. Partial asymptotic stabilization can be achieved by adding dissipation- 
emulating terms to the control input. 
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42.5 Motion Planning 


Motion planning problems for nonholonomic and underactuated systems can be characterized along a 
number of dimensions. Is the system drift-free or not? Are there configuration obstacles? What form do 
control constraints take? Is it more important to find a solution that minimizes time or energy, or is it 
more important to find a satisficing motion plan quickly? 

Motion planning for nonholonomic and underactuated systems is an active research area, and in this 
section we only briefly describe a few approaches. More can be found in [2,11,13,17,18] and references 
therein. 


42.5.1 Numerical Optimal Control 


This approach typically involves choosing a finite-dimensional parameterization of the control history 
u:[0, T] — U and solving for these design variables to minimize a cost function while satisfying equality 
and inequality constraints. Example constraints include control limits, terminal state conditions, and 
perhaps obstacle constraints, sampled at equally spaced points in time. The nonlinear optimization 
problem is typically solved by an algorithm that takes advantage of the gradient (and perhaps Hessian) 
of the cost function and constraints with respect to the design variables. A typical algorithm choice is 
Sequential Quadratic Programming. A variant of the approach is to dispense with the objective function 
and to simply cast the problem as a nonlinear root-finding problem. These approaches can be applied 
to general underactuated systems with drift. The success of these methods depends on problem-specific 
characteristics, such as the quality of the initial guess to the gradient-based algorithm, the number of local 
minima in the design space, and the method used to calculate gradients. 


42.5.2 Optimal Control of the Heisenberg System 


Some motion planning problems admit an analytical optimal solution. One such problem is to drive the 
Heisenberg system from an initial state [x1, x2, z] T —[0,0, 0]? toa goal state [0,0,a > 0] T in time T while 
minimizing the cost functional 


1 Tr 
: (uj + u5) dt. 
An equivalent formulation is the following: Minimize the integral 
1 T 
5 i (x7 + <3) dt 
among all curves q(t) joining q(0) = [0, 0, 0} to q(T) = [0, 0, a]! that satisfy the constraint 
z= xX] Sue x1 X2. 


Any solution must satisfy the Euler-Lagrange equations for the Lagrangian with a Lagrange multiplier 
inserted: 


L(x, 1, 2,252.2, ds) = 4 (RT +43) + (2 — x41 + x14). 
The corresponding Euler-Lagrange equations are given by 


%, —2hdn = 0, (42.106) 
X. + 20x, = 0, (42.107) 


h=0. (42.108) 


Control of Nonholonomic and Underactuated Systems 42-31 


We can show the solution of the optimal control problem is given by choosing initial conditions such that 
(0)? + 2(0)* = 2a/T? and with the trajectory in the x) x2-plane given by the circle 


eA A ee -—1  —~sin(2nt/T) Ba 


xy(t)| 2x | sin(2xt/T) cos(2nt/T) —1 (0) (42.109) 


and with z given by 


(t) ta a. 2nt 
z(t) = = sin ; 
T 20 T 
A motion planner to an arbitrary final state [x)q, xq, zal" could simply use the two controls to drive the 


first two configuration variables directly to the goal state, followed by the optimal trajectory to accomplish 
the remaining necessary motion in z. More details can be found in [2] and [9]. 


42.5.3 Motion Planning for the Generalized Heisenberg System 


The approach above can be generalized to chained-form systems [13,18] and the generalized Heisenberg 
system (Equations 42.76 and 42.77). 

As with the Heisenberg system, for the generalized Heisenberg system we first use the controls to drive 
the system directly to the desired x states, and then use sinusoidal controls to accomplish a net motion in 
Y while producing zero net motion in x. The idea is to proceed along loops in x-space, which gradually 
drives one through Y-space. 

We choose the control law 


uj = >> aig sin kt +) biz coskt, k= Tees) (42.110) 
k k 


where aj, and bj, are real numbers. Since x; = uj, integration gives 
dik bik. 
se ar a a (42.111) 


where Cj is a constant depending on the initial value of x. 
Substituting these equations for x;(t) and u;(t) into Equation 42.77 and integrating yields 


21 
¥y(2n) = > ; (bia — Bjeai) + Yi(0), (42.112) 
k 


since all integrals except those of the squares of cosine and sine vanish. Under this input, x has zero net 
motion. More details can be found in [2]. 


42.5.4 Search-Based Methods 


Configuration obstacles can be accounted for by constraints in a numerical optimization or by using search 
algorithms from computer science [13,17]. Search-based methods for motion planning for underactuated 
systems usually involve choosing a small number of “representative” controls from the control set. One 
or more of these controls are integrated forward for a short time At from the initial state xp to create a 
new set of reached states V. Trajectories that intersect obstacles are discarded, and the remaining states 
in % are put in a list sorted by the cost of reaching the state. The first state on the list is then removed, 
assigned to be xo, and the process continues until xo is in a neighborhood of the goal state. To reduce the 
planning time or improve the plans, heuristics can be designed for choosing the controls, pruning states 
that are “close” to previously reached states, and assigning the cost of states. This approach is applied to 
parallel parking a car-like robot in Figure 42.7. The cost trades off the length of the path and the number 
of control changes (e.g., cusps). 
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FIGURE 42.7 A car parallel parking. 


42.5.5 Path Transformation Methods 


If the system is kinematic and STLC, one can first plan a path among obstacles assuming the system has 
no nonholonomic constraints, using one of the many collision-free motion planning methods developed 
in the robotics literature. This plan can then be transformed into a motion plan that satisfies the velocity 
constraints [13]. Typically this transformation proceeds by recursively choosing a segment of the initial 
path and attempting to replace it with a path chosen by a local path planner. The local planner takes the 
initial and final configurations qo and q¢ of the segment and chooses a path satisfying the nonholonomic 
constraints, but ignoring the obstacles. The replacement occurs if the new path segment is collision-free. 
If not, the initial segment is subdivided again. To guarantee success of the transformation process, the 
local path planner must have the property that the length of the chosen path goes to zero as || q¢— qo || 
goes to zero. The final feasible path produced by the transformation could be further post-processed to 
reduce some measure of the cost of the path. 


42.5.6 Kinematic Reductions 


For dynamic underactuated systems with kinematic reductions yielding STLKC, trajectory planning from 
one equilibrium state to another can be reduced to the problem of path planning using the kinematic 
reductions, followed by time-scaling of the path to produce a trajectory [11-13]. This decoupling of the 
trajectory planning problem reduces the computational complexity by turning the constrained motion 
planning problem on the 2n-dimensional state space into path planning on the n-dimensional config- 
uration space. In addition, time-optimal time scaling algorithms can be used to turn the paths into fast 
feasible trajectories. Because the kinematic reductions take account of the underactuation constraints, the 
only constraints on path speed come from the actuator limits. 

An example is shown in Figure 42.8 for the 3R robot of Figure 42.4. The robot arm has no 
actuator at the third link and is described by Li2 = 0.3m, 71,23 = 0.15m, mj; = {2.0, 1.0,0.5}kg, I; = 
{0.02, 0.01, 0.004125} kgm? with actuator limits |u| < 20 Nm, |u2| < 10 Nm. As mentioned earlier, the 
two decoupling vector fields are translation along the third link and rotation of the third link about its 
center of percussion with respect to the third joint. Using just these motions, we can use a path planner 
to plan point-to-point motions. Figure 42.8 shows an example path consisting of four segments along the 
decoupling vector fields. Also shown is the time-optimal time scaling, which allows the robot to follow 
the path as quickly as possible subject to actuator limits. Because the only common velocity between the 
two decoupling vector fields is zero velocity, the robot must come to a stop at the transitions between 
vector fields. 
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Path u, and uy (Nm) vs. time (s) 


FIGURE 42.8 Four path segments and their time scalings for a 3R robot without an actuator at the third joint. Note 
that the time-optimal time scalings saturate one actuator at all times. 


42.5.7 Differentially Flat Systems 


Differentially flat and dynamically feedback linearizable systems have a structure that makes motion plan- 
ning (in the absence of control and configuration constraints such as obstacles) particularly simple [23]. 
For a differentially flat system with a state x and u € R”, there exists a set of m functions z;,i=1,...,m, 
of the state, the control, and its derivatives, 


z(x,u,u,...,u), i=1,...,m, 
such that the states and control inputs can be expressed as functions of z and its time-derivatives: 


x= 0(z,Z,... 2), 


u=W(z,Z,... 2), 


The functions z; are known as the flat outputs. Armed with a set of flat outputs, the problem of 
finding a feasible trajectory (x(t), u(t)), x(0) = xo, x(t¢) = x, t € [0, te] for the underactuated system is 


42-34 Control System Advanced Methods 


transformed to the problem of finding a curve z(t) satisfying constraints on z(0),zZ(0),... ,2)(0) and 
z(t), Z(t), ..., 2 (te) specified by xo and x¢. In other words, the problem of finding a trajectory satisfying 
the underactuation constraints becomes the relatively simple algebraic problem of finding a curve to fit 
the start and end constraints on z. Any curve z(t) maps directly to a consistent pair of state and control 
histories x(t) and u(t). 

The flat outputs for mechanical systems are often a function of configuration variables only, and 
sometimes are just the location of particular points on the system. Unfortunately, there is no systematic 
way to determine if a system is differentially flat, or what the flat outputs for a system are. Many important 
systems have been shown to be differentially flat, however, such as chained-form systems, cars, and cars 
pulling trailers. 

Asan example, consider a vertical rolling disk, ignoring the configuration variable . The configuration 
is q =[x, y, 61", and the controls are the forward velocity v and the turning rate w (Figure 42.2). The flat 
outputs are simply z} = x and z) = y. The state and controls can be derived from the flat outputs and 
their derivatives as follows: 


aT 

Zz 

[xy]? = [zrezstan-! =] : (42.113) 
1 


oe oe . T 
2122 — Z| Z 
[vw]? =| +27 +23, ~S—S" | (42.114) 
+z 


The orientation $ and the turning control w are not well defined as a function of the flat outputs when 
the linear velocity of the disk is zero. 

Now we would like to find a feasible trajectory from qo = [0, 0, 0] to qe= [1 1, 0]. Since there are six 
state variables in the specification of the start and goal points, there are six constraints on the flat outputs 
z and their derivatives at the beginning and end of motion. These constraints can be written 


z1(0)=0, z(0)=0, 
a(t) =1, a(t) =1, 
22(0) =0, 
22 (te) = 0, 


where the last two constraints indicate that the initial and final motion of the unicycle must be along the 
x-axis, indicating that the wheel is oriented with the x-axis. The simplest polynomial functions of time 
that have enough free coefficients to satisfy these constraints are 


z(t) =ay+ait, 
zo(t) = bo + byt + bot? + b3t?. 


Setting the time of motion t¢ = 1 and using the constraints to solve for the polynomial coefficients, we 
obtain 


z(t) =t, (42.115) 
z(t) = 3t* — 20. (42.116) 


The state and control can be obtained from Equations 42.113 and 42.114. The unicycle motion is shown 
in Figure 42.9. 

In fitting a curve z(t), we must choose a family of curves with enough degrees of freedom to satisfy 
the initial and terminal constraints. We may choose a family of curves with more degrees of freedom, 
however, and use the extra degrees of freedom to, individually or severally, (1) satisfy bounds on the 
control u(t), (2) avoid obstacles in the configuration space, or (3) minimize a cost function. Incorporating 
these conditions in the calculation of z(t) typically requires resorting to numerical optimization methods. 
A good way to generate an initial guess for the optimization is to solve exactly for a minimal number of 
coefficients to satisfy the initial and terminal constraints, setting the other coefficients to zero. 
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FIGURE 42.9 A feasible path for the unicycle from [0, 0, 0] to [1, 1, 0]", and the controls. 
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43.1 Introduction 


Stability theory plays a central role in systems theory and engineering. It deals with the system’s behavior 
over a long time period. There are several ways to characterize stability. For example, we may characterize 
stability from an input-output viewpoint, by requiring the output of a system to be “well-behaved” in 
some sense, whenever the input is well behaved. Alternatively, we may characterize stability by studying 
the asymptotic behavior of the state of the system near steady-state solutions, like equilibrium points or 
periodic orbits. 

In this chapter we introduce Lyapunov’s method for determining the stability of equilibrium points. 
Lyapunov laid the foundation of this method over a century ago, but of course the method as we use 
it today is the result of intensive research efforts by many engineers and applied mathematicians. The 
attractive features of the method include a solid theoretical foundation, the ability to conclude stability 
without knowledge of the solution (no extensive simulation effort), and an analytical framework that 
makes it possible to study the effect of model perturbations and to design feedback control. Its main 
drawback is the need to search for an auxiliary function that satisfies certain conditions. 


43.2 Stability of Equilibrium Points 


We consider a nonlinear system represented by the state model 
x =f (x) (43.1) 


where the components of the n-dimensional vector f(x) are locally Lipschitz functions of x, defined for 
all x in a domain D C R”. A function f(x) is locally Lipschitz at a point xo if it satisfies the Lipschitz 
condition ||f (x) — f(y) || < L||x — y|| for all x, y in some neighborhood of xo, where L is a positive constant 
and ||x|| = ,/x7 +x5+---+x2. The Lipschitz condition guarantees that Equation 43.1 has a unique 
solution for a given initial state x(0). Suppose x € D is an equilibrium point of Equation 43.1; that is, 
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f(x) = 0. Whenever the state of the system starts at x it will remain at x for all future time. Our goal is to 
characterize and study the stability of x. For convenience, we take x = 0. There is no loss of generality in 
doing so because any equilibrium point x can be shifted to the origin via the change of variables y = x — x. 
Therefore, we shall always assume that f(0) = 0, and study stability of the origin x = 0. 

The equilibrium point x = 0 of Equation 43.1 is stable, if for each ¢ > 0, there is 8 = 8(e) > 0 such 
that ||x(0)|| <8 implies that ||x(t)|| <, for all t > 0. It is asymptotically stable, if it is stable and 5 can 
be chosen such that ||x(0)|| < 5 implies that x(t) converges to the origin as t tends to infinity. When the 
origin is asymptotically stable, the region of attraction (also called region of asymptotic stability, domain 
of attraction, or basin) is defined as the set of all points x such that the solution of Equation 43.1 that 
starts from x at time ¢ = 0, approaches the origin as t tends to oo. When the region of attraction is the 
whole space, we say that the origin is globally asymptotically stable. A stronger form of asymptotic stability 
arises when there exist positive constants c, k, and ) such that the solutions of Equation 43.1 satisfy the 
inequality 

IIx(2)Il <kIlx(O)Ile-™, Vt>0 (43.2) 


for all ||x(0)|| < c. In this case, the equilibrium point x = 0 is said to be exponentially stable. It is said to 
be globally exponentially stable if the inequality is satisfied for any initial state x(0). 


43.2.1 Linear Systems 


For the linear time-invariant system 
x= Ax (43.3) 


the stability properties of the origin can be characterized by the location of the eigenvalues of A. The 
origin is stable if and only if all the eigenvalues of A satisfy Re[;] < 0 and for every eigenvalue with 
Re[A;] = 0 and algebraic multiplicity qj > 2, rank(A — ;I) = n— qi, where n is the dimension of x and 
qi is the multiplicity of 4; as a zero of det(AJ — A). The origin is globally exponentially stable if and 
only if all eigenvalues of A have negative real parts; that is, A is a Hurwitz matrix. For linear systems, 
the notions of asymptotic and exponential stability are equivalent because the solution is formed of 
exponential modes. Moreover, due to linearity, if the origin is exponentially stable, then the inequality of 
Equation 43.2 will hold for all initial states. 


43.2.2 Linearization 


Suppose the function f(x) of Equation 43.1 is continuously differentiable in a domain D containing the 
origin. The Jacobian matrix [Of /0x] is an n x n matrix whose (i,j) element is Of;/0x;. Let A be the 
Jacobian matrix evaluated at the origin x = 0. It can be shown that 


f(x) =[A+ G(x)]x, where lim G(x) =0 


This suggests that in a small neighborhood of the origin we can approximate the nonlinear system x = f (x) 
by its linearization about the origin x = Ax. Indeed, we can draw conclusions about the stability of the 
origin as an equilibrium point for the nonlinear system by examining the eigenvalues of A. The origin 
of Equation 43.1 is exponentially stable if and only if A is Hurwitz. It is unstable if Re[;] > 0 for one or 
more of the eigenvalues of A. Linearization fails when Re[i;] < 0 for all i, with Re[X;] = 0 for some i, for 
in this case we cannot draw a conclusion about the stability of the origin of Equation 43.1. 


43.2.3 Lyapunov Method 


Let V(x) be a continuously differentiable scalar function defined in a domain D C R” that contains the 
origin. The function V(x) is said to be positive definite if V(0) = 0 and V(x) > 0 for x 4 0. It is said to 
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be positive semidefinite if V(x) > 0 for all x. A function V(x) is said to be negative definite or negative 
semidefinite if —V(x) is positive definite or positive semidefinite, respectively. The derivative of V along 
the trajectories of Equation 43.1 is given by 


; "av av 
Val=) 0a t= 5 fe) 


Ox, 
i=1 ~~ 


where [OV /0x] is a row vector whose ith component is OV /O0x;. 

Lyapunov’s stability theorem states that the origin is stable if there is a continuously differentiable 
positive-definite function V(x) so that V(x) is negative semidefinite, and it is asymptotically stable if V(x) 
is negative definite. 

A function V(x) satisfying the conditions for stability is called a Lyapunov function. The surface 
V(x) =c, for some c > 0, is called a Lyapunov surface or a level surface. Using Lyapunov surfaces, 
Figure 43.1 makes the theorem intuitively clear. It shows Lyapunov surfaces for decreasing constants 
c3 > C2 > cy > 0. The condition V < 0 implies that when a trajectory crosses a Lyapunov surface V(x) = c, 
it moves inside the set Q, = {V(x) < c} and can never come out again, since V <0 on the boundary 
V(x) =c. When V <0, the trajectory moves from one Lyapunov surface to an inner Lyapunov surface 
with a smaller c. As c decreases, the Lyapunov surface V(x) = c shrinks to the origin, showing that the 
trajectory approaches the origin as time progresses. If we only know that V < 0, we cannot be sure that 
the trajectory will approach the origin, but we can conclude that the origin is stable since the trajectory 
can be contained inside any ¢ neighborhood of the origin by requiring the initial state x(0) to lie inside a 
Lyapunov surface contained in that neighborhood. 

When V(x) is only negative semidefinite, we may still conclude asymptotic stability of the origin if we 
can show that no solution can stay identically in the set {V(x) = 0}, other than the zero solution x(t) = 0. 
Under this condition, V(x(t)) must decrease toward 0, and consequently x(t) converges to zero as t tends 
to infinity. This extension of the basic theorem is known as the invariance principle. 

Lyapunov functions can be used to estimate the region of attraction of an asymptotically stable origin, 
that is, to find sets contained in the region of attraction. Let V(x) be a Lyapunov function that satisfies the 
conditions of asymptotic stability over a domain D. For a positive constant c, let (2, be the component 
of {V(x) < c} that contains the origin in its interior. The properties of V guarantee that, by choosing c 
small enough, Q2, will be bounded and contained in D. Then, every trajectory starting in Q, remains in 
Q2,, and approaches the origin as t > oo. Thus, 22, is an estimate of the region of attraction. If D = R” 
and V(x) is radially unbounded, that is, ||x|| — oo implies that V(x) — oo, then any point x € R” can be 
included in a bounded set (2, by choosing c large enough. Therefore, the origin is globally asymptotically 
stable if there is a continuously differentiable, radially unbounded function V(x) such that for all x € R", 


FIGURE 43.1 Lyapunov surfaces V(x) = c; with cz > c, > cy. 
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V(x) is positive definite and V(x) is either negative definite or negative semidefinite, but no solution can 
stay identically in the set {V (x) = 0} other than the zero solution x(t) = 0. 

Lyapunov’s method is a powerful tool for studying the stability of equilibrium points. However, there 
are two drawbacks of the method that we should be aware of. First, there is no systematic method for 
finding a Lyapunov function for a given system. Second, the conditions of the method are only sufficient; 
they are not necessary. Failure of a Lyapunov function candidate to satisfy the conditions for stability or 
asymptotic stability does not mean that the origin is not stable or asymptotically stable. 


43.2.4 Time-Varying Systems 


Equation 43.1 is time-invariant because f does not depend on t. The more general time-varying system is 
represented by 


k=f(t.x) (43.4) 


In this case, we may allow the Lyapunov function candidate V to depend on t. Let V(t, x) be a continuously 
differentiable function defined for all t > 0 and all x € D. The derivative of V along the trajectories of 
Equation 43.4 is given by 

OV 


2 OV 
V(t,x) = aE + Bul 9) 


If there are positive-definite functions W(x), W2(x), and W3(x) such that 


W(x) < V(t, x) < W(x) (43.5) 
V(t, x) < —Ws(x) (43.6) 


for all t > 0 and all x € D, then the origin is uniformly asymptotically stable, where “uniformly” indicates 
that the e-8 definition of stability and the convergence of x(t) to zero are independent of the initial 
time fp. Such uniformity annotation is not needed with time-invariant systems since the solution of a 
time-invariant state equation starting at time to depends only on the difference t — to, which is not the 
case for time-varying systems. If the inequalities of Equations 43.5 and 43.6 hold globally and Wj (x) 
is radially unbounded, then the origin is globally uniformly asymptotically stable. If Wy (x) = ky ||x||*, 
W(x) = kg||x||%, and W3(x) = k3||x||* for some positive constants k,, kz, k3, and a, then the origin is 
exponentially stable. 


43.2.5 Perturbed Systems 


Consider the system 
x=f (t,x) + g(t,x) (43.7) 


where f and g are continuous in ¢ and locally Lipschitz in x, for all t > 0 and x € D, in which D C R" 
is a domain that contains the origin x = 0. Suppose f(t, 0) = 0 and g(t,0) =0 so that the origin is an 
equilibrium point of Equation 43.7. We think of the system of Equation 43.7 as a perturbation of the 
nominal system 


k=f(t.x) (43.8) 


The perturbation term g(t, x) could result from modeling errors, uncertainties, or disturbances, which 
exist in any realistic problem. Ina typical situation, we do not know g(t, x), but we know some information 
about it, like knowing an upper bound on ||g(t, x) ||. Suppose the nominal system has an exponentially 
stable equilibrium point at the origin, what can we say about the stability of the origin as an equilibrium 
point of the perturbed system? A natural approach to address this question is to use a Lyapunov function 
for the nominal system as a Lyapunov function candidate for the perturbed system. 
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Let V(t, x) be a Lyapunov function that satisfies 


cillxll? < V(t,x) < ealle|l? (43.9) 
OV. av 
ae + Bul *) <—c3||x\| (43.10) 
OV 
| —]] < call (43.11) 
Ox 


for all x € D for some positive constants c), cz, cz, and cq. Suppose the perturbation term g(t, x) satisfies 
the linear growth bound 


g(t, x)Il < yilxll, VWt>=0, VxeD (43.12) 


where y is a nonnegative constant. We use V as a Lyapunov function candidate to investigate the stability 
of the origin as an equilibrium point for the perturbed system. The derivative of V along the trajectories 
of Equation 43.7 is given by 


dV OV av 
V(t, x) = — + —fit, = galt, 
(x)= a+ a fbx) + 3 alba) 


The first two terms on the right-hand side are the derivative of V(t, x) along the trajectories of the nominal 
system, which is negative definite and satisfies the inequality of Equation 43.10. The third term, [OV /Ox]g, 
is the effect of the perturbation. Using Equations 43.10 through 43.12, we obtain 


; OV 
V(t,x) < —cs||x||? + | | \Ig(t, x) || < —csllxll? + caylloxll? 


If y < c3/c4, then 
V(t, x) < —(c3—yea)llxll?, (cz — yea) > 0 


which shows that the origin is an exponentially stable equilibrium point of Equation 43.7. 


43.3 Examples and Applications 


Example 43.1: 
A simple pendulum moving in a vertical plane can be modeled by the state equation 


xy =xX2 


xX = —asinx, — bx2 


where a > 0, b> 0, and x) is the angle between the pendulum and the vertical line through the 
pivot point. The case b = 0 is an idealized frictionless pendulum. To find the equilibrium points, we 
set X1 = X72 = 0 and solve for x; and x2. The first equation gives x7 = 0 and the second one gives 
sin x; = 0. Thus, the equilibrium points are located at (nz, 0), for n =0,+1,+2,... The pendulum 
has only two equilibrium positions corresponding to the equilibrium points (0,0) and (1,0). The 
other equilibrium points are repetitions of these two positions that correspond to the number of full 
swings the pendulum would make before it rests at one of the two equilibrium positions. 
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To start with, let us investigate the stability of the equilibrium points by linearization. The Jacobian 
matrix is given by 


an Of 

of | dx Oxm| [ 0 1 

ax af, ar oe =| 
Ax, Oxy 


To investigate the stability of the origin, we evaluate the Jacobian matrix at x = 0. 


eee 
dese? “Pot eB 


The eigenvalues of A are \1,2 = (— + /b? — 4a) /2. For all positive values of a and b, the eigenvalues 
satisfy Re[\] < 0. Hence, the equilibrium point at the origin is exponentially stable. In the absence of 
friction (b = 0), both eigenvalues are on the imaginary axis. In this case, we cannot determine the stability 
of the origin through linearization. To investigate the stability of (x, 0), we evaluate the Jacobian at this 
point. This is equivalent to performing a change of variables z} = x; — 1, Z2 = x2 to shift the equilibrium 
point to the origin, and then evaluating the Jacobian [Of /0z] at z = 0. 


of " b 1 
xj =11,x2=0 a —b 


As 
Ox 
The eigenvalues of A are \12 = (—btv b +4a) /2. For all a>0 and b > 0, there is, at least, one 
eigenvalue with positive real part. Hence, (1, 0) is unstable. 
Let us now use Lyapunov’s method to study the stability of the origin. As a Lyapunov function candidate, 
we use the energy of the pendulum, which is defined as the sum of its potential and kinetic energies, 
namely, 


Bal iT 1 
vin)= f asiny dy + 5x3 =a(1—cosxi) + 5x3 
0 


The reference of the potential energy is chosen such that V(0) = 0. The function V(x) is positive definite 
over the domain —2n < x; < 2m. The derivative of V along the trajectories of the system is given by 


V(x) = ax sin x) +22%). = — bx 


When friction is neglected (b = 0), V(x) =0 and we can conclude that the origin is stable. Moreover, 
V(x) is constant during the motion of the system. Since V(x) = c forms a closed contour around x = 0 
for small c > 0, we see that the trajectory will be confined to one such contour and will not approach the 
origin. Hence the origin is not asymptotically stable. On the other hand, in the case with friction (b > 0), 
V(x) = —bx} < 0 is negative semidefinite and we can conclude that the origin is stable. Note that V(x) is 
only negative semidefinite and not negative definite because V(x) = 0 for x2 = 0 irrespective of the value 
of x,. Therefore, we cannot conclude asymptotic stability using Lyapunov’s stability theorem. Here comes 
the role of the invariance principle. Consider the set {V(x) = 0} = {xm = 0}. Suppose that a solution of 
the state equation stays identically in this set. Then 


x2(t)=0 > (t)=0 = sinx)(t)=0 


Hence, on the segment —1m < x, < m of the x2 = 0 line, the system can maintain the V(x) = 0 condition 
only at the origin x = 0. Noting that the solution is confined to Q, (the component {V(x) < c} that 
contains the origin) and for sufficiently small c, Q, C {—m < x, < m}, we conclude that no solution 
can stay forever in the set Q. {x2 = 0} other than the trivial solution x(t) =0. Hence, the origin is 
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asymptotically stable. We can also estimate the region of attraction by 2, with c < 2a to ensure that Q, 
is bounded and contained in the strip {—m < x, < mt}. 


Example 43.2: 


Consider the system 
X1 =X2 
X2 = —91(%1) — g2(x2) 
where g, and gp are locally Lipschitz and satisfy 
gi(0)=0, ygily)>0, Vy #0, i=1,2 


and ie g1(z) dz — ov, as |y| > oo. The system has an isolated equilibrium point at the origin. It 
can be viewed as a generalized pendulum equation with g2(x2) as the friction term. Therefore, a 
Lyapunov function candidate may be taken as the energy-like function 


XI 1 2 
vi = | oi) dy + 5, 
0 2 


which is positive definite in R2 and radially unbounded. The derivative of V(x) along the trajectories 
of the system is given by 


V(x) = 1 (x4)x2 + X2l—91 (x1) — ga(xa)] = —x2g2(x2) < 0 


Thus, V(x) is negative semidefinite. Note that V(x) =0 implies x2g2(x2) = 0, which implies x2 = 0. 
Therefore, the only solution that can stay identically in the set {x € R2 | x7 = 0} is the zero solution 
x(t) = 0. Thus, the origin is globally asymptotically stable. 


Example 43.3: 


The second-order system 
x = —X2 
Xo = x x7 — 1)xg 


has a unique equilibrium point at the origin. Linearization at the origin yields the matrix 


_ OF = = 
veo «LI 7-1 


Ox 
which is Hurwitz. Hence, the origin is exponentially stable. By viewing the nonlinear system x = f(x) 
as a perturbation of the linear system x = Ax, we can find a Lyapunov function for the linear system 
and use it as a Lyapunov function candidate for the nonlinear system. For the linear system x = Ax 
with Hurwitz matrix A, a quadratic Lyapunov function is given by V(x) = x! Px, where P is the solution 
of the Lyapunov equation 


A 


PA+A'P=-Q 


for any positive-definite symmetric matrix Q. This is so because the solution P of the Lyapunov equa- 
tion is a positive-definite symmetric matrix and the derivative of V(x) = x! Px along the trajectories 
of x = Ax is —x! Qx. For our example, taking Q = / results in 


1.5 —0.5 
P=[3 1 | 


Now weuse V(x) = x! Pxasa Lyapunov function candidate for the nonlinear system. The perturbation 
term f(x) — Ax satisfies the linear growth bound of Equation 43.12 and the constant y can be made 
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arbitrarily small by limiting the analysis to a sufficiently small neighborhood of the origin. This is so 
because f(x) — Ax = G(x)x, where limy_, 9 G(x) = 0. Therefore, from the earlier analysis of perturbed 
systems we know that V(x) is a Lyapunov function for the nonlinear system in some neighborhood 
of the origin. Let us estimate the region of attraction. The function V(x) is positive definite for all 
x. We need to determine a domain D about the origin where V(x) is negative definite and a set 
Qe = {V(x) < c} C D. The set Qc- is bounded for any c > 0. We are interested in the largest set Qc 
that we can determine, that is, the largest value for the constant c. The derivative of V(x) along the 
trajectories of the system is given by 


V(x) = (xp + x3) — xF.x2 (x1 — 2x2) 


The right-hand side of V(x) is written as the sum of two terms. The first term, —|[x||2, is the con- 
tribution of the linear part Ax, while the second term is the contribution of the nonlinear term 
f(x) — Ax. Using the inequalities |x; — 2x2| < V/5||x|| and |x;x2| < s(x, we see that V(x) satisfies 
the inequality V(x) < —||x||? + (5/2)||x||4. Hence V(x) is negative definite in the region {||x|| <r}, 
where r2 = 2/5. We would like to choose a positive constant c such that {V(x) < c} isa subset of this 
region. Since x! Px > min (P)||x||2, we can choose ¢ < Amin(P)r2. Using Amin(P) = 0.69, we choose 
c=0.615 < 0.69(2/+/5) = 0.617. The set {V(x) < 0.615} is an estimate of the region of attraction. 


Example 43.4: 


Consider the time-varying system 


x1 =—X] — g(t)x2 
+) = X1—xX2 


where g(t) is continuously differentiable and satisfies 0 < g(t) < k and g < g(t) for all t>0. The 
system has an equilibrium point at the origin. Consider a Lyapunov function candidate V(t, x) = 
ey + [1+ g(t)]x2. The function V satisfies the inequalities 


XG SV tx) < xP +1 + XG 
The derivative of V along the trajectories of the system is given by 
V = 2x? + 2xyxy — [2 + 2g(t) — g(t) x3 
Using the bound on g(t), we have 2 + 2g(t) — g(t) = 2 + 2g(t) — g(t) = 2. Therefore, 


-1 


V < 2x? + 2x1x9 — 2x3 = —x7 ie ; 


x= —x! Qx 


The matrix Q is positive definite. Hence, the origin is globally exponentially stable. 


43.3.1 Feedback Stabilization 
Consider the nonlinear system 


x =f (x, u) 
y = h(x) 
where x is an n-dimensional state, u is an m-dimensional control input, and y is a p-dimensional measured 


output. Suppose f and h are continuously differentiable in the domain of interest, and f (0, 0) = 0, h(0) = 0 
so that the origin is an open-loop equilibrium point and the output y vanishes at the origin. Suppose 
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we want to design an output feedback controller to stabilize the origin, that is, to make the origin 
an asymptotically stable equilibrium point of the closed-loop system. We can pursue the design via 
linearization. The linearization of the system about the point (x = 0, u = 0) is given by 


x= Ax+ Bu 
y=Cx 
h 
renee pe Ree 
Ox x=0,u=0 Ou x=0,u=0 Ox x=0 


Assuming that (A, B) is stabilizable and (A, C) is detectable, that is, uncontrollable and unobservable 
eigenvalues, if any, have negative real parts, we can design a dynamic output feedback controller 


z=Fz+Gy, 
u=Hz+ ky, 


where z is a q-dimensional vector, such that the matrix 


4a|4+BKC BH 
“| GC F 


is Hurwitz. When the feedback controller is applied to the nonlinear system it results in a system of order 
(n+ q), whose linearization at the origin is the matrix A. Hence, the origin of the closed-loop system 
is exponentially stable. By solving the Lyapunov equation PA + A’P = —Q for some positive-definite 
matrix Q, we can use V = XP, where X = [x z]", asa Lyapunov function for the closed-loop system, 
and we can estimate the region of attraction of the origin, as illustrated in Example 43.3. 


43.4 Defining Terms 


Asymptotically stable equilibrium point: A stable equilibrium point with the additional feature that all 
trajectories starting at nearby points approach the equilibrium point as time approaches infinity. 

Equilibrium point: A constant solution of x =f (t,x). For the time-invariant system x = f(x), equilib- 
rium points are the real solutions of the equation 0 = f(x). 

Exponentially stable equilibrium point: An asymptotically stable equilibrium point with the additional 
feature that the norm of the state is bounded by an exponentially decaying function of time 
whose amplitude is proportional to the norm of the difference between the initial state and the 
equilibrium point. 

Globally asymptotically stable equilibrium point: An asymptotically stable equilibrium point where the 
region of attraction is the whole space. 

Hurwitz matrix: A square real matrix is Hurwitz if all its eigenvalues have negative real parts. 

Linearization: Approximation of the nonlinear state equation in the vicinity of an equilibrium point by 
a linear state equation, obtained by calculating the Jacobian matrix of the right-hand side at the 
equilibrium point. 

Lipschitz condition: A condition imposed on a function f(x) to ensure that it has a finite slope. For a 
vector-valued function, it takes the form ||f(x) — f(y)|| < L|lx — y|| for some positive constant L. 

Locally Lipschitz function: A function f(x) is locally Lipschitz at a point if it satisfies the Lipschitz 
condition in the neighborhood of that point. 

Lyapunov equation: A linear algebraic matrix equation of the form PA + A? P = —Q, where A and Qare 
real square matrices. When Q is symmetric and positive definite, the equation has a (unique) 
positive-definite solution P if and only if A is Hurwitz. 
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Lyapunov function: A scalar positive-definite function of the state whose derivative along the trajectories 
of the system is negative semidefinite. 

Lyapunov surface: A set of the form V(x) =c where V(x) is a Lyapunov function and c is a positive 
constant. 

Negative (semi-) definite function: A scalar function of a vector argument V(x) is negative (semi-) 
definite if V(0) = 0 and V(x) < 0 (< 0) for all x 4 0 in some neighborhood of x = 0. 

Positive (semi-) definite function: A scalar function ofa vector argument V (x) is positive (semi-) definite 
if V(0) = 0 and V(x) > 0 (= 0) for all x 4 0 in some neighborhood of x = 0. 

Positive-definite matrix: A symmetric real square matrix P is positive definite if the quadratic form 
V(x) = x" Px is a positive definite function. Equivalently, P is positive definite if and only if all 
its eigenvalues are positive. 

Region of attraction: For an asymptotically stable equilibrium point, the region of attraction is the set of 
all points with the property that the trajectories starting at these points asymptotically approach 
the equilibrium point. It is an open connected set that contains the equilibrium point in its 
interior. 

Stable equilibrium point: An equilibrium point where all solutions can be confined to an e-neighborhood 
of the point be constraining the initial states to belong to a 8-neighborhood. 


Reference 


1. Khalil, H-K. 2002. Nonlinear Systems, 3rd Edition, Prentice-Hall, Upper Saddle River, NJ. 


Further Reading 


The presentation of Lyapunov stability is based on the textbook by Khalil (see [1]). For further information 
on Lyapunov stability, the reader is referred to Chapters 4, 8, and 9 of Khalil’s book. Chapter 4 covers the 
basic Lyapunov theory. Chapter 8 covers more advanced topics, including the use of the center manifold 
theorem when linearization fails. Chapter 9 covers the stability of perturbed systems. 

Other engineering textbooks where Lyapunov stability is emphasized include Vidyasagar, M. 2002. 
Nonlinear Systems Analysis, classic Ed., SIAM Philadelphia, PA; Slotine, J-J. and Li, W. 1991. Applied 
Nonlinear Control, Prentice-Hall, Englewood Cliffs; Sastry, S. 1999. Nonlinear Systems: Analysis, Stability, 
and Control, Springer, New York. 

For a deeper look into the theoretical foundation of Lyapunov stability, there are excellent references, 
including Rouche, N., Habets, P., and Laloy, M. 1977. Stability Theory by Lyapunov’s Direct Method, 
Springer-Verlag, New York; Hahn, W. 1967. Stability of Motion, Springer-Verlag, New York; Krasovskii, 
N.N. 1963. Stability of Motion, Stanford University Press, Palo Alto, CA. 

Control journals often include articles where Lyapunov’s method is used in system analysis or control 
design. Examples are the IEEE Transactions on Automatic Control and the IFAC Journal Automatica. 
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44.1 Introduction 


A common task for an engineer is to design a system that reacts to stimuli in some specific and desirable 
way. One way to characterize appropriate behavior is through the formalism of input-output stability. In 
this setting a notion of well-behaved input and output signals is made precise and the question is posed: 
do well-behaved stimuli (inputs) produce well-behaved responses (outputs)? 

General input-output stability analysis has its roots in the development of the electronic feedback 
amplifier of H.S. Black in 1927 and the subsequent development of classical feedback design tools for lin- 
ear systems by H. Nyquist and H.W. Bode in the 1930s and 1940s, all at Bell Telephone Laboratories. These 
latter tools focused on determining input-output stability of linear feedback systems from the characteris- 
tics of the feedback components. Generalizations to nonlinear systems were made by several researchers in 
the late 1950s and early 1960s. The most notable contributions were those of G. Zames, then at M.LT., I.W. 
Sandberg at Bell Telephone Laboratories, and V.M. Popov. Indeed, much of this chapter is based on the 
foundational ideas found in [5,7,10], with additional insights drawn from [6]. A thorough understanding 
of nonlinear systems from an input-output point of view is still an area of ongoing and intensive research. 

The strength of input-output stability theory is that it provides a method for anticipating the qualitative 
behavior of a feedback system with only rough information about the feedback components. This, in turn, 
leads to notions of robustness of feedback stability and motivates many of the recent developments in 
modern control theory. 


44.2 Systems and Stability 


Throughout our discussion of input-output stability, a signal is a “reasonable” (e.g., piecewise continuous) 
function defined on a finite or semi-infinite time interval, i.e., an interval of the form [0, T) where T is 
either a strictly positive real number or infinity. In general, a signal is vector-valued; its components 
typically represent actuator and sensor values. A dynamical system is an object which produces an output 
signal for each input signal. 


44-1 
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To discuss stability of dynamical systems, we introduce the concept of a norm function, denoted || - ||, 
which captures the “size” of signals defined on the semi-infinite time interval. The significant properties 
of a norm function are that 1) the norm of a signal is zero if the signal is identically zero, and is a strictly 
positive number otherwise, 2) scaling a signal results in a corresponding scaling of the norm, and 3) the 
triangle inequality holds, ie., ||) + u2|| < ||u1|| + ||u2||. Examples of norm functions are the p-norms. 
For any positive real number p > 1, the p-norm is defined by 


o0 3 
I|ullp = (/ io) (44.1) 


n 
oe u;. For p = 00, we define 


i=1 


where | - | represents the standard Euclidean norm, ie., |u| = 


IIulloo == sup |u(t)]. (44.2) 
10 

The oo-norm is useful when amplitude constraints are imposed on a problem, and the 2-norm is of 
more interest in the context of energy constraints. The norm of a signal may very well be infinite. We 
will typically be interested in measuring signals which may only be defined on finite time intervals or 
measuring truncated versions of signals. To that end, given a signal u defined on [0, T) and a strictly 
positive real number 1, we use u, to denote the truncated signal generated by extending u onto [0, oo) by 
defining u(t) = 0 for t > T, if necessary, and then truncating, i.e., u; is equal to the (extended) signal on 
the interval [0, t] and is equal to zero on the interval (t, 00). 

Informally, a system is stable in the input-output sense if small input signals produce correspondingly 
small output signals. To make this concept precise, we need a way to quantify the dependence of the 
norm of the output on the norm of the input applied to the system. To that end, we define a gain function 
as a function from the nonnegative real numbers to the nonnegative real numbers which is continuous, 
nondecreasing, and zero when its argument is zero. For notational convenience we will say that the “value” 
of a gain function at oo is oo. A dynamical system is stable (with respect to the norm || - ||) if there is a 
gain function y which gives a bound on the norm of truncated output signals as a function of the norm 
of truncated input signals, i.e., 


IIycl] Sy |ucll), for all t. (44.3) 


In the very special case when the gain function is linear, i.e., there is at most an amplification by a constant 
factor, the dynamical system is finite gain stable. The notions of finite gain stability and closely related 
variants are central to much of classical input-output stability theory, but in recent years much progress 
has been made in understanding the role of more general (nonlinear) gains in system analysis. 

The focus of this chapter will be on the stability analysis of interconnected dynamical systems as 
described in Figure 44.1. The composite system in Figure 44.1 will be called a well-defined interconnection 


Nn 


x 


FIGURE 44.1 Standard feedback configuration. 
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if it is a dynamical system with (41) as input and (; 
2 


”) as output, i.e., given an arbitrary input signal 
2 


(i) a signal (; ) exists so that, for the dynamical system %, the input d, + y2 produces the output 
2 2, 


y, and, for the dynamical system X2, the input d,+ y, produces the output y2. To see that not every 
interconnection is well-defined, consider the case where both X and X2 are the identity mappings. 


In this case, the only input signals G) for which an output 6 ') can be found are those for which 
2 2 


d, +d, =0. The dynamical systems which make up a well-defined interconnection will be called its 
feedback components. 

For stability of well-defined interconnections, it is not necessary for either of the feedback components 
to be stable nor is it sufficient for both of the feedback components to be stable. On the other hand, 
necessary and sufficient conditions for stability of a well-defined interconnection can be expressed in 
terms of the set of all possible input-output pairs for each feedback component. To be explicit, following 
are some definitions. For a given dynamical system & with input signals u and output signals y, the set 
of its ordered input-output pairs (u, y) is referred to as the graph of the dynamical system and is denoted 
Gy. When the input and output are exchanged in the ordered pair, i-e., (y, u), the set is referred to as the 
inverse graph of the system and is denoted G4. Note that, for the system in Figure 44.1, the inverse graph 
of X2 and the graph of 2} lie in the same Cartesian product space called the ambient space. We will use 
as norm on the ambient space the sum of the norms of the coordinates. 

The basic observation regarding input-output stability for a well-defined interconnection says, in 
informal terms, that if a signal in the inverse graph of Xp is near any signal in the graph of X| then it must 
be small. To formalize this notion, we need the concept of the distance to the graph of X) from signals x 
in the ambient space. This (truncated) distance is defined by 


d,(x,Gp,) = pe I(x — z)zl|- (44.4) 


=] 


Theorem 44.1: Graph Separation Theorem 


A well-defined interconnection is stable if, and only if, a gain function y exists which gives a bound on the 
norm of truncated signals in the inverse graph of Xz as a function of the truncated distance from the signals 
to the graph of Xj, i.e., 

xe Gh, => ||x||<y (d, (x, Gy,)) , forall. (44.5) 


In the special case where y is a linear function, the well-defined interconnection is finite gain stable. 


The idea behind this observation can be understood by considering the signals that arise in the closed 
loop which belong to the inverse graph of X23, i-e., the signals (y2, y) + dz). (Stability with these signals taken 
as output is equivalent to stability with the original outputs.) Notice that, for the system in Figure 44.1, 
signals in the graph of 2) have the form (yz + d1, 1). Consequently, signals x € Ss, and z € Gy,, which 
satisfy the feedback equations, also satisfy 


(x — z)q = (d), —do) (44.6) 


and 
I(x — z)x|] = Idi, d2)qI| (44.7) 


for truncations within the interval of definition. If there are signals x in the inverse graph of X2 with large 
truncated norm but small truncated distance to the graph of 1, i.e., there exists some z € Gy, andt > 0 
such that ||(x — z),|| is small, then we can choose (dj, d2) to satisfy Equation 44.6 giving, according to 
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Equation 44.7, a small input which produces a large output. This contradicts our definition of stability. 
Conversely, if there is no z which is close to x, then only large inputs can produce large x signals and thus 
the system is stable. 

The distance observation presented above is the unifying idea behind the input-output stability criteria 
applied in practice. However, the observation is rarely applied directly because of the difficulties involved 
in exactly characterizing the graph of a dynamical system and measuring distances. Instead, various 
simpler conditions have been developed which constrain the graphs of the feedback components to 
guarantee that the graph of X, and the inverse graph of X> are sufficiently separated. There are many 
such sufficient conditions, and, in the remainder of this chapter, we will describe a few of them. 


44.3 Practical Conditions and Examples 
44.3.1 The Classical Small Gain Theorem 


One of the most commonly used sufficient conditions for graph separation constrains the graphs of 
the feedback components by assuming that each feedback component is finite gain stable. Then, the 
appropriate graphs will be separated if the product of the coefficients of the linear gain functions is 
sufficiently small. For this reason, the result based on this type of constraint has come to be known as the 
small gain theorem. 


Theorem 44.2: Small Gain Theorem 


If each feedback component is finite gain stable and the product of the gains (the coefficients of the linear 
gain functions) is less than one, then the well-defined interconnection is finite gain stable. 


Figure 44.2 provides the intuition for the result. If we were to draw an analogy between a dynamical 
system and a static map whose graph is a set of points in the plane, the graph of X; would be constrained 
to the darkly shaded conic region by the finite gain stability assumption. Likewise, the inverse graph of 
&2 would be constrained to the lightly shaded region. The fact that the product of the gains is less than 
one guarantees the positive aperture between the two regions and, in turn, that the graphs are separated 
sufficiently. 

To apply the small gain theorem, we need a way to verify that the feedback components are finite gain 
stable (with respect to a particular norm) and to determine their gains. In particular, any linear dynamical 
system that can be represented with a real, rational transfer function G(s) is finite gain stable in any of the 
p-norms if, and only if, all of the poles of the transfer function have negative real parts. A popular norm 
to work with is the 2-norm. It is associated with the energy of a signal. For a single-input, single-output 
(SISO) finite gain stable system modeled by a real, rational transfer function G(s), the smallest possible 
coefficient for the stability gain function with respect to the 2-norm, is given by 


y := sup |G(ja)|. (44.8) 


For multi-input, multioutput systems, the magnitude in Equation 44.8 is replaced by the maximum 
singular value. In either case, this can be established using Parseval’s theorem. For SISO systems, the 
quantity in Equation 44.8 can be obtained from a quick examination of the Bode plot or Nyquist plot for 
the transfer function. If the Nyquist plot of a stable SISO transfer function lies inside a circle of radius 
y centered at the origin, then the coefficient of the 2-norm gain function for the system is less than or 
equal to y. 


Input-Output Stability 44-5 


1/? 


Inverse 
graph of 5 


Graph of 


FIGURE 44.2 Classical small gain theorem. 


More generally, consider a dynamical system that can be represented by a finite dimensional ordinary 
differential equation with zero initial state: 


x=f(x,u), x(0)=0 and y=A(x,u). (44.9) 


Suppose that f has globally bounded partial derivatives and that positive real numbers ¢) and £2 exist 
so that 
|h(x, u)| < Ci|x| + £o[ ul. (44.10) 


Under these conditions, if the trajectories of the unforced system with nonzero initial conditions, 
x=f(x,0), x(0) =x, (44.11) 


satisfy 
|x(t)| < kexp(—Xt)|xol, (44.12) 


for some positive real number k and } and any x, € R", then the system (Equation 44.9) is finite gain 
stable in any of the p-norms. This can be established using Lyapunov function arguments that apply to 
the system (Equation 44.11). The details can be found in the textbooks on nonlinear systems mentioned 
later. 


Example 44.1: 
Consider a nonlinear control system modeled by an ordinary differential equation with state x <« R”, 
input v € R™ and disturbance d; ¢ R™: 

xX=f(x,v+d)). (44.13) 


Suppose that f has globally bounded partial derivatives and that a control v = a(x) can be found, 
also with a globally bounded partial derivative, so that the trajectories of the system 


x =F(x,a(x)), x(0)=Xo (44.14) 


44-6 Control System Advanced Methods 


satisfy the bound 
x(t)| < k exp(—At)|xo| (44.15) 


for some positive real numbers k and » and for all x. € R". As mentioned above, for any function h 
satisfying the type of bound in Equation 44.10, this implies that the system 


x =f(x,a(x)+d)), x(0)=0 and y=h(x,d)) (44.16) 


has finite 2-norm gain from input d; to output y. We consider the output 
0 
yes 5, ft (x) +d) (44.17) 


which satisfies the type of bound in Equation 44.10 because a and f both have globally bounded 
partial derivatives. 

We will show, using the small gain theorem, that disturbances dj with finite 2-norm continue to 
produce outputs y with finite 2-norm even when the actual input v to the process is generated by 
the following fast dynamic version of the commanded input a(x): 

ez = Az+Bla(x))+d2, z(0) = —A7~'Ba(x(0)) 


(44.18) 
v=Cz. 


Here, € is a small positive parameter, the eigenvalues of A all have strictly negative real part (thus A is 
invertible), and —CA~'B =I. This system may represent unmodeled actuator dynamics. 

To see the stability result, we will consider the composite system in the coordinates x and t= 
Z+A7|Ba(x). Using the notation from Figure 44.1, 


x =f(x,a(x)+u7), x(0)=0 
2A (44.19) 
y1 =A | Bax), 
and 
t=e Attu, t=0 
29: (44.20) 
y2 = Ce, 
with the interconnection conditions 
Uj =y2+d,, and ug=y,+e~'dp. (44.21) 


Of course, if the system is finite gain stable with the inputs d; and «— ‘dp, then it is also finite gain 
stable with the inputs d; and d2. We have already discussed that the system 41 in Equation 44.19 
has finite 2-norm gain, say y;. Now consider the system &2 in Equation 44.20. It can be represented 
with the transfer function 


G(s) = C(s!—e—'A)—!, 
=€C(es|—A)—', (44.22) 


=: €G(es). 
Identifying G(s) = C(s! — A)", we see that, if 
Y2 := sup o(G(jw)), (44.23) 
wo 


then 
sup o(G(jw)) = ey. (44 24) 
@ 


1 
We conclude from the small gain theorem that, if ¢ < ——, then the composite system (Equa- 
v1¥2 
tions 44.19 through 44 21), with inputs d; and d2 and outputs y; = A—'Ba(x) and y2 = Ct, is finite 


gain stable. 
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44.3.2 The Classical Passivity Theorem 


Another very popular condition used to guarantee graph separation is given in the passivity theorem. 
For the most straightforward passivity result, the number of input channels must equal the number of 
output channels for each feedback component. We then identify the relative location of the graphs of the 
feedback components using a condition involving the integral of the product of the input and the output 
signals. This operation is known as the inner product, denoted (., -). In particular, for two signals u and y 
of the same dimension defined on the semi-infinite interval, 


(uy) 2= / : u! (t)y(t) dt. (44.25) 
0 


Note that (u, y) = (y, u) and (u, u) = ||u| ee A dynamical system is passive if, for each input-output pair 
(u, y) and each t > 0, (uz, yr) > 0. The terminology used here comes from the special case where the input 
and output are a voltage and a current, respectively, and the energy absorbed by the dynamical system, 
which is the inner product of the input and output, is nonnegative. 

Again by analogy to a static map whose graph lies in the plane, passivity of a dynamical system can 
be viewed as the condition that the graph is constrained to the darkly shaded region in Figure 44.3, 
ie., the first and third quadrants of the plane. This graph and the inverse graph of a second system 
would be separated if, for example, the inverse graph of the second system were constrained to the 
lightly shaded region in Figure 44.3, ie., the second and fourth quadrants but bounded away from 
the horizontal and vertical axes by an increasing and unbounded distance. But, this is the same as 
asking that the graph of the second system followed by the scaling “—1,” ie., all pairs (u,—y), be 
constrained to the first and third quadrants, again bounded away from the axes by an increasing and 
unbounded distance, as in Figure 44.4a. For classical passivity theorems, this region is given a linear 
boundary as in Figure 44.4b. Notice that, for points (u,, yo) in the plane, if uo -y. > €(u2 + y2) then 
(Uo, Yo) is in the first or third quadrant, and (€)~!|u,| = |yo| = €|uo| as in Figure 44.4b. This leads to 


Graph of >, 


Inverse 
graph of >» 


FIGURE 44.3 General passivity-based interconnection. 
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FIGURE 44.4 Different notions of input and output strict passivity. 


the following stronger version of passivity. A dynamical system is input and output strictly passive 
if a strictly positive real number € exists so that, for each input-output pair (u,y) and each t > 0, 
(utes yx) = €(Ilucll3 + Ilyell3). 

There are intermediate versions of passivity which are also useful. These correspond to asking for an 
increasing and unbounded distance from either the horizontal axis or the vertical axis but not both. For 
example, a dynamical system is input strictly passive if a strictly positive real number € exists so that, for 
each input-output pair (u, y) and each t > 0, (uz, yx) = €||uz||3. Similarly, a dynamical system is output 
strictly passive if a strictly positive real number € exists so that, for each input-output pair (u, y) and 
each t > 0, (uz, x) = €llyz||5. It is worth noting that input and output strict passivity is equivalent to 
input strict passive plus finite gain stability. This can be shown with standard manipulations of the inner 
product. Also, the reader is warned that all three types of strict passivity mentioned above are frequently 
called “strict passivity” in the literature. 

Again by thinking of a graph of a system as a set of points in the plane, output strict passivity is the 
condition that the graph is constrained to the darkly shaded region in Figure 44.5, i.e., the first and third 
quadrants with an increasing and unbounded distance from the vertical axis. To complement such a 
graph, consider a second dynamical system which, when followed by the scaling “—1,” is also output 
strictly passive. Such a system has a graph (without the “—1” scaling) constrained to the second and 
fourth quadrants with an increasing and unbounded distance from the vertical axis. In other words, its 
inverse graph is constrained to the lightly shaded region of Figure 44.5, i.e., to the second and fourth 
quadrants but with an increasing and unbounded distance from the horizontal axis. The conclusions 
that we can then draw, using the graph separation theorem, are summarized in the following passivity 
theorem. 


Theorem 44.3: Passivity Theorem 


If one dynamical system and the other dynamical system followed by the scaling “—1” are 
¢ both input strictly passive, OR 
¢ both output strictly passive, OR 


* respectively, passive and input and output strictly passive, 


then the well-defined interconnection is finite gain stable in the 2-norm. 
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FIGURE 44.5 Interconnection of output strictly passive systems. 


To apply this theorem, we need a way to verify that the (possibly scaled) feedback components are 
appropriately passive. For stable SISO systems with real, rational transfer function G(s), it again follows 
from Parseval’s theorem that, if 


Re G(jw) = 0, 


for all real values of w, then the system is passive. If the quantity Re G(jw) is positive and uniformly 
bounded away from zero for all real values of w, then the linear system is input and output strictly passive. 
Similarly, if ¢ > 0 exists so that, for all real values of w, 


Re G(jw — €) > 0, (44.26) 


then the linear system is output strictly passive. So, for SISO systems modeled with real, rational transfer 
functions, passivity and the various forms of strict passivity can again be easily checked by means of a 
graphical approach such as a Nyquist plot. 

More generally, for any dynamical system that can be modeled with a smooth, finite dimensional 
ordinary differential equation, 


x=f(x)+g(x)u, x(0)=0 


baie (44.27) 


if a strictly positive real number € exists and a nonnegative function V: R” > R> 0 with V(0)=0 
satisfying 
OV T 
By FO < —€h’ (x)h(x), (44.28) 
x. 


and 
oY ge) =h" (x), (44.29) 


then the system is output strictly passive. With € = 0, the system is passive. Both of these results are 
established by integrating V over the semi-infinite interval. 
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Example 44.2: 


(This example is prompted by the work in Berghuis and Nijmeijer, Syst. Control Lett., 1993, 21, 289- 
295.) Consider a “completely controlled dissipative Euler-Lagrange” system with generalized “forces” 
F, generalized coordinates q, uniformly positive definite “inertia” matrix /(q), Rayleigh dissipation 
function R(q) and, say positive, potential V(q) starting from the position gg. Let the dynamics of the 
system be given by the Euler-Lagrange-Rayleigh equations, 


OL OL OL 
asa: ; ni Reeeae ; . Fr a VE ys 
aq (9,9) dq (9,9) + dq (q) 
q(0)=qd, q(0)=0, (44.30) 
where L is the Lagrangian 
; V3 ; 
Lia.) = 5 qa! (q)q—V(q). (44.31) 


Along the solution of Equation 44.30, 


OL OL ol or OR 
: . Fr a 


i = g = P 44.32 
pat eqs ris dg Dg ( ) 
= F =(0q'! —|F --~-|q 44.33 
a) | a ( q (9)4) | ag |4 ( ) 
a R ; R 
aS yelee eo eva eee (44 34) 
oq oq 
We will suppose that « > 0 exists so that 
OR: a) 
wae > ; 44.35 
Dg (9)q = €/q| ( ) 
Now let Vy be a function so that the modified potential 
Vn =V+Vq (44.36) 
has a global minimum at q = qq, and let the generalized “force” be 
OV 
pis Liars, (44.37) 
0q 


We can see that the system (Equation 44.30) combined with Equation 44.37, having input Fy, and 
output q, is output strictly passive by integrating the derivative of the defined Hamiltonian, 


H= 547g) a + Vlg) = b+ 2V4Vg. (44.38) 
Indeed the derivative is 
H= Fn = 7a | g (44.39) 
and, integrating, for each t 
([Fm. - 5540" | i] = H(t) — H(0). (44.40) 


Since H > 0, H(0) = 0 and Equation 44.35 holds 
(Fm 4x) = €llacll}- (44.41) 


Using the notation from Figure 44.1, let X1 be the system (Equations 44.30 and 44.37) with input 
Fm and output q. Let Xz be any system that, when followed by the scaling “—1,” is output strictly 
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passive. Then, according to the passivity theorem, the composite feedback system as given in Fig- 
ure 44.1 is finite gain stable using the 2-norm. One possibility for 22 is minus the identity mapping. 
However, there is interest in choosing =z followed by the scaling “—1” as a linear, output strictly 
passive compensator which, in addition, has no direct feed-through term. The reason is that if, d2 in 
Figure 44.1 is identically zero, we can implement Xz with measurement only of g and without g. In 
general, 


: 1, 

G(s)q = G(s)s (4) = G(s)s(q — qq), (44.42) 
and the system G(s)s is implementable if G(s) has no direct feed-through terms. To design an output 
strictly passive linear system without direct feed-through, let A be a matrix having all eigenvalues 
with strictly negative real parts so that, by a well-known result in linear systems theory, a positive 


definite matrix P exists satisfying 
ATP+PA=-l. (44.43) 


Then, for any B matrix of appropriate dimensions, the system modeled by the transfer function, 
G(s) = —B! P(sI— A)~'B, (44.44) 
followed by the scaling “—1,” is output strictly passive. To see this, consider a state-space realization 


X=Ax+Bu_ x(0)=0 


T (44.45) 
y =B' Px, 
and note that 
sor, 
x" Px=—x!x +2x'PBu (44.46) 
=-x'x + 2y'u. (44.47) 
But, with Equation 44.45, for some strictly positive real number c, 
2cyly <x!'x. (44.48) 
So, integrating Equation 44.47 and with P positive definite, for all t, 
(YorUr) > cllycll3. (44.49) 
As a point of interest, one could verify that 
G(s)s = —B! PA(s| — A) 'B— B" PB. (44.50) 


44.3.3 Simple Nonlinear Separation Theorems 


In this section we illustrate how allowing regions with nonlinear boundaries in the small gain and passivity 
contexts may be useful. First we need a class of functions to describe nonlinear boundaries. A proper 
separation function is a function from the nonnegative real numbers to the nonnegative real numbers 
which is continuous, zero at zero, strictly increasing and unbounded. The main difference between a gain 
function and a proper separation function is that the latter is invertible, and the inverse is another proper 
separation function. 


44.3.3.1 Nonlinear Passivity 


We will briefly discuss a definition of nonlinear input and output strict passivity. To our knowledge, this 
idea has not been used much in the literature. The notion replaces the linear boundaries in the input 
and output strict passivity definition by nonlinear boundaries as in Figure 44.4a. A dynamical system 
is nonlinearly input and output strictly passive if a proper separation function p exists so that, for each 
input-output pair (u, y) and each t > 0, (uz, yr) > |[urll2eC|ucll2) + Ilycll2eC|yrl|2). (Note that in the 
classical definition of strict passivity, e(¢) = €¢ for all ¢ > 0.) 
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Theorem 44.4: Nonlinear Passivity Theorem 


If one dynamical system is passive and the other dynamical system followed by the scaling “— 1” is nonlinearly 
input and output strictly passive, then the well-defined interconnection is stable using the 2-norm. 


Example 44.3: 


Let ©, bea single integrator system, 


XxX, =u, X1(0)=0 
¥1=%1.- 


(44.51) 


This system is passive because 
1 2 7 d | 2 : 
0 < 5x1(t) =A ae x(t) ar =f yy (t)uy(t) dt = (yi,,U1,). (44.52) 


Let Xz bea system which scales the instantaneous value of the input according to the energy of the 
input: 


xQ= ie X2(0) =0 


1 (44.53) 
=-—U - 
- (as) 


This system followed by the scaling “—1” is nonlinearly strictly passive. To see this, first note that 
x2(t) = [lu2, 15 (4454) 


which is a nondecreasing function of t. So, 


w= [ (ca) 
aed =I T+ [xp (925 


i 

> 2(t) dt, 
= (Sees ae raps) Ih Ne (44.55) 
= IIu2, 113 

(le + luz, 119° 3s] 

Now we can define 
cee (44.56) 
i?) i — 1 +05’ . 


which is a proper separation function, so that 
(—y2,,U2,) = 2p(||u2, Il2)|lu2, Il2- (44.57) 


Finally, note that 


v 1 
WwaclB= f° w32 at ¢ lua, (44.58) 
o (1 +x95(t)) 


so that 
(—y2,,42,) = olllu2,Il2)||42, 12 + oll yz, 12) l¥2,l2- (44.59) 
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FIGURE 44.6 Nonlinear small gain theorem. 


The conclusion that we can then draw from the nonlinear passivity theorem is that the interconnec- 
tion of these two systems: 
2 


’ 


| 
x ==) +42) (35) +h X2 = (x1 +.d2 
Peale (44.60) 


1 
=X, and = —(x; + d2) | ———~= 
Y=X Yaa OI » (as) 


is stable when measuring input (d;,d2) and output (y7, yz) using the 2-norm. 


44,3.3.2 Nonlinear Small Gain 


Just as with passivity, the idea behind the small gain theorem does not require the use of linear boundaries. 
Consider a well-defined interconnection where each feedback component is stable but not necessarily 
finite gain stable. Let y, be a stability gain function for X, and let y2 be a stability gain function for Xp. 
Then the graph separation condition will be satisfied if the distance between the curves (¢, y1(¢)) and 
(y2(&), &) grows without bound as in Figure 44.6. This is equivalent to asking whether it is possible to add 
to the curve (¢, y1(¢)) in the vertical direction and to the curve (y2(&), €) in the horizontal direction, by an 
increasing and unbounded amount, to obtain new curves (6 yils) + o(¢)) and (y2(€) + o(&), 3) where 0 
is a proper separation function, so that the modified first curve is never above the modified second curve. 
If this is possible, we will say that the composition of the functions y; and y> is a strict contraction. To say 
that a curve (¢, y1(¢)) is never above a second curve (/2(&), &) is equivalent to saying that V1 (2(¢)) <¢ or 
yo(¥i()) < ¢ for all ¢ > 0. (Equivalently, we will write Vy; 0 jz < Id or 2 0 jy; < Id.) So, requiring that the 
composition of y; and y2 is a strict contraction is equivalent to requiring that a strictly proper separation 
function p exists so that (yj + p) o(y2 + p) < Id (equivalently (y2 + p) o (yi + p) < Id). This condition 
was made precise in [3]. (See also [2].) Note that it is not enough to add to just one curve because it is 
possible for the vertical or horizontal distance to grow without bound while the total distance remains 
bounded. Finally, note that, if the gain functions are linear, the condition is the same as the condition that 
the product of the gains is less than one. 


44-14 Control System Advanced Methods 


Theorem 44.5: Nonlinear Small Gain Theorem 


If each feedback component is stable (with gain functions y; and y2) and the composition of the gains is a 
strict contraction, then the well-defined interconnection is stable. 


To apply the nonlinear small gain theorem, we need a way to verify that the feedback components are 
stable. To date, the most common setting for using the nonlinear small gain theorem is when measuring 
the input and output using the oo-norm. For a nonlinear system which can be represented by a smooth, 
ordinary differential equation, 


x=f(x,u), x(0)=0, and y=A(x,u), (44.61) 


where h(0, 0) = 0, the system is stable (with respect to the oo-norm) if there exist a positive definite and 
radially unbounded function V : R” + R= 0, a proper separation function yy, and a gain function Y so 
that 


oY flew) < Wx) +7(ud. (44.62) 


Since V is positive definite and radially unbounded, additional proper separation functions a and @ exist 
so that 
a(|x|) < V(x) < a(|x]). (44.63) 


Also, because h is continuous and zero at zero, gain functions ), and ¢, exist so that 
|h(x, u)| < dx(lx1) + u(|u)). (44.64) 
Given all of these functions, a stability gain function can be computed as 
V=oxoa odo oVt+ou. (44.65) 


For more details, the reader is directed to [8]. 


Example 44.4: 


Consider the composite system, 


x =Ax+Bsat(z+d}), x(0)=0 


; (44.66) 
Z=—z-+e(exp(|x|+d2)—1), z(0)=0, 


where x € R",z € R, the eigenvalues of A all have strictly negative real part, ¢ is a small parameter, and 
sat(s) = sgn(s) min{|s|, 1}. This composite system is a well-defined interconnection of the subsystems 
xX = Ax + Bsat(u;), x(0)=0 
D1: (44.67) 
1 = |X| 
and 
z=-z+e(exp(u2)—1), 2(0)=0 
x2 F (44.68) 
y2 =2Z. 
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A gain function for the 1 system is the product of the oo-gain for the linear system 


xX=Ax+Bu, x(0)=0 


(44.69) 
y=x, 
which we will call 71, with the function sat(s), i.e., for the system 1, 
IYIloo S Y1sat(||U4 |loo). (44.70) 
For the system 22, 
IIZlloo < lel (exp (IIU2Iloo) — 1). (44.71) 


The distance between the curves (¢, y1Sat(c)) and (|| (exp (&) — 1) ,&) must grow without bound. 
Graphically, one can see that a necessary and sufficient condition for this is that 


1 
«| << —___—.. (44.72) 
I exp(y1) —1 


44.3.4 General Conic Regions 


There are many different ways to partition the ambient space to establish the graph separation condition 
in Equation 44.5. So far we have looked at only two very specific sufficient conditions, the small gain 
theorem and the passivity theorem. The general idea in these theorems is to constrain signals in the graph 
of X within some conic region, and signals in the inverse graph of X outside of this conic region. Conic 
regions more general than those used for the small gain and passivity theorems can be generated by using 
operators on the input-output pairs of the feedback components. 

Let C and R be operators on truncated ordered pairs in the ambient space, and let y be a gain function. 
We say that the graph of X) is inside Cone(C, R, y) if, for each (u, y) =: z belonging to the graph of X), 


C(z-)I| < yIR(Z)I|), for all t. (44.73) 


On the other hand, we say that the inverse graph of X2 is strictly outside Cone(C, R, y) ifa proper separation 
function p exists so that, for each (y, u) =: x belonging to the inverse graph of Xp, 


IIC(xz)|] = yo Ud + p)(IRGx)I[) + o(|lrl[), for all t. (44.74) 


We will only consider the case where the maps C and R are incrementally stable, i.e., a gain function y 
exists so that, for each x; and x2 in the ambient space and all t, 
[]C(x1,) — C(x2, JI] <= villa, — x2, 1) 


i (44.75) 
[|R(x1,) — R(x2, || < vllx1, — x2, 11). 


In this case, the following result holds. 


Theorem 44.6: Nonlinear Conic Sector Theorem 


If the graph of X is inside CONE(C, R, y) and the inverse graph of X2 is strictly outside CONE(C, R, y), then 
the well-defined interconnection is stable. 


When y and p are linear functions, the well-defined interconnection is finite gain stable. 

The small gain and passivity theorems we have discussed can be interpreted in the framework of the 
nonlinear conic sector theorem. For example, for the nonlinear small gain theorem, the operator C is a 
projection onto the second coordinate in the ambient space, and R is a projection onto the first coordinate; 
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FIGURE 44.7 Instantaneous sector. 


y is the gain function y,, and the small gain condition guarantees that the inverse graph of X2 is strictly 
outside of the cone specified by this C, R and y. 
In the remaining subsections, we will discuss other useful choices for the operators C and R. 


44,3.4.1 The Classical Conic Sector (Circle) Theorem 


For linear SISO systems connected to memoryless nonlinearities, there is an additional classical result, 
known as the circle theorem, which follows from the nonlinear conic sector theorem using the 2-norm 
and taking 


C (u,y) =ytcu 
R (u,y) =ru r>0 (44.76) 
y(o) =o. 


Suppose is a memoryless nonlinearity which satisfies 
Id(u, t) + cul < |ru| for all tf, u. (44.77) 


Graphically, the constraint on $ is shown in Figure 44.7. (In the case shown, c > r > 0.) We will use the 
notation Sector[—(c + r), —(c — r)] for the memoryless nonlinearity. It is also clear that the graph of lies 
in the ConE(C, R, y) with C, R, y defined in Equation 44.76. For a linear, time invariant, finite dimensional 
SISO system, whether its inverse graph is strictly outside of this cone can be determined by examining 
the Nyquist plot of its transfer function. The condition on the Nyquist plot is expressed relative to a disk 
Dz, in the complex plane centered on the real axis passing through the points on the real axis with real 
parts —1/(c+1r) and —1/(c—r) as shown in Figure 44.8. 


Theorem 44.7: Circle Theorem 


Let r>0, and consider a well-defined interconnection of a memoryless nonlinearity belonging to 
SecToR[—(c+ 1), —(c — r)] with a SISO system having a real, rational transfer function G(s). If 
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FIGURE 44.8 A disc in the complex plane. 


* r>c, G(s) is stable and the Nyquist plot of G(s) lies in the interior of the disc D,, or 

* r=c, G(s) is stable and the Nyquist plot of G(s) is bounded away and to the right of the vertical line 
passing through the real axis at the value —1/(c +r), or 

* r<c, the Nyquist plot of G(s) (with Nyquist path indented into the right-half plane) is outside of 
and bounded away from the disc D,,, and the number of times the plot encircles this disc in the 
counterclockwise direction is equal to the number of poles of G(s) with strictly positive real parts, 


then the interconnection is finite gain stable. 


Case 1 is similar to the small gain theorem, and case 2 is similar to the passivity theorem. We will now 
explain case 3 in more detail. Let n(s) and d(s) represent, respectively, the numerator and denominator 
polynomials of G(s). Since the point (—1/c, 0) is inside the disc D,,,, it follows, from the assumption of the 
theorem together with the well-known Nyquist stability condition, that all of the roots of the polynomial 
d(s) + cn(s) have negative real parts. Then y = G(s)u= N(s)D(s)~!u where 


D(s) := SS and N(s):= — (44.78) 
and, by taking z = D(s)~!u, we can describe all of the possible input-output pairs as 
(u,y) = (D(s)z, N(s)z) ‘ (44.79) 
Notice that D(s) + cN(s) = 1, so that 
\|u+ cy||2 = ||ZIl2- (44.80) 


To put a lower bound on this expression in terms of ||u||2 and ||y||2, to show that the graph is strictly 
outside of the cone defined in Equation 44.76, we will need the 2-norm gains for systems modeled by the 
transfer functions N(s) and D(s). We will use the symbols yy and yp for these gains. The condition of 
the circle theorem guarantees that yy < r~!. To see this, note that 


_ G(s) 
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implying 
= G(ja) 
YN = ae i eGGe) GG) | : (44.82) 
But 
|1+c¢ G(jw)|? — r?|G(jo)|* 
= (c Re {G(jo)} +1)’ +2 Im? {G(jo)} — 7? Re® {G(jo)} — 1? Im? {G(jo)}, (44.93) 


‘ 2 72 
=(c?—?r?) (Re (@tio)} +55) ie =7) Im? {G(jo)} — s— 5 


yp 


Setting the latter expression to zero defines the boundary of the disc D,,,. Since the expression is positive 
outside of this disc, it follows that yy < ro}, 
Returning to the calculation initiated in Equation 44.80, note that yy <1r~ 


positive real number € exists so that 


| implies that a strictly 


(1—eyp)yy' = r+2¢. (44.84) 


So, 


\lu+ cyll2 = ||zlle = 1 — eyp)Ilzll2 + evpllzll2; 
> (1—eyp)yq'Ilyll2 + €llull, (44.85) 
> (r +6)llyllo+(\lulls + lyll2)- 


We conclude that the inverse graph of the linear system is strictly outside of the CONE(C, R, y) as defined 
in Equation 44.76. 

Note, incidentally, that N(s) is the closed loop transfer function from d; to y; for the special case 
where the memoryless nonlinearity satisfies @(u) = —cu. This suggests another way of determining sta- 
bility: first make a preliminary loop transformation with the feedback —cu, changing the original linear 
system into the system with transfer function N(s) and changing the nonlinearity into a new nonlin- 
earity ¢ satisfying |(u, f)| < r|u|. Then apply the classical small gain theorem to the resulting feedback 
system. 


Example 44.5: 


Let 
175 


(s—1)(s +4)?” 


The Nyquist plot of G(s) is shown in Figure 44.9. Because G(s) has one pole with positive real part, only 
the third condition of the circle theorem can apply. A disc centered at —8.1 on the real axis and with 
radius 2.2 can be placed inside the left loop of the Nyquist plot. Such a disc corresponds to the values 
c = 0.293 and r = 0.079. Because the Nyquist plot encircles this disc once in the counterclockwise 
direction, it follows that the standard feedback connection with the feedback components G(s) and 
a memoryless nonlinearity constrained to the Sector[—0.372,—0.214] is stable using the 2-norm. 


44.3.4.2 Coprime Fractions 


Typical input-output stability results based on stable coprime fractions are corollaries of the conic sector 
theorem. For example, suppose both ©; and Xz are modeled by transfer functions G;(s) and Gy(s). 
Moreover, assume stable (in any p-norm) transfer functions N;, D,, Nj, D1, Nz and D2 exist so that Dj, 
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FIGURE 44.9 The Nyquist plot for G(s) in Example 44.5. 


D> and Dy are invertible, and 


G, = ND; (44.87) 
Id = D) D2 — N,N. 


Let C(u, y) = Dy(s)y —Nj(s)u, which is incrementally stable in any p-norm, let R(u, y) = 0, and let 
y =0. Then, the graph of X is inside and the inverse graph of X2 is strictly outside CoNE(C, R, y) and 
thus the feedback loop is finite gain stable in any p-norm. 

To verify these claims about the properties of the graphs, first recognize that the graph of X; can be 
represented as 


Gs; = (Di(s)z, _Nj(s)z) (44.88) 


where z represents any reasonable signal. Then, for signals in the graph of 2), 


Cc (Di (s)z:; Ni (s)zx) — Di(s)Ny (s)z, — NA (s)D,(s)z, = 0. (44.89) 
Conversely, for signals in the inverse graph of Xp, 
|| C (N2(s)zx, D2(s)zx) |] = |] Di (s)D2(s)ze — Ni (s)N2(s)ze|| (44,90) 


= |[Zcll 2 € | (N2(s)zx,Da(s)z-) | 


for some strictly positive real number e. The last inequality follows from the fact that Dz and N> are finite 
gain stable. 


Example 44.6: 


(This example is drawn from the work in Potvin, M-J., Jeswiet, J., and Piedboeuf, J.-C. , Trans. 
NAMRI/SME 1994, XXII, pp 373-377.) Let X1 represent the fractional Voigt—-Kelvin model for the 
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relation between stress and strain in structures displaying plasticity. For suitable values of Young's 
modulus, damping magnitude, and order of derivative for the strain, the transfer function of Dj is 


1 
gi(s) 


—t Tamu EE Vs 
1 
Integral feedback control, g2(s) = — eG may be used for asymptotic tracking. Here 
1 1+ 4/5 
Ni(s)= ——, Dy(s) = Me 
s+1 s+1 
(44.91) 
No(s) s+] s(s+1) 
s) = -——_——__., s) = ——_—__.. 
z (ses) 2 SESS) 


It can be shown that these fractions are stable linear operators, and thereby incrementally stable 
in the 2-norm. (This fact is equivalent to proving nominal stability and can be shown using Nyquist 
theory.) Moreover, it is easy to see that D} D2 — N;N> = 1 so that the feedback loop is stable and 
finite gain stable. 


44.3.4.3 Robustness of Stability and the Gap Metric 


It is clear from the original graph separation theorem that, if a well-defined interconnection is stable, ie., 
the appropriate graphs are separated in distance, then modifications of the feedback components will not 
destroy stability if the modified graphs are close to the original graphs. 

Given two systems , and %, define 3(q x) = a if a is the smallest number for which 


xEGs, = > d,(x,Gy,) < allx||, for all t. 


The quantity 3(-,-) is called the “directed gap” between the two systems and characterizes basic neighbor- 
hoods where stability as well as closed-loop properties are preserved under small perturbations from the 
nominal system Xj to a nearby system &. 

More specifically, if the interconnection of (21, Xz) is finite gain stable, we define the gain By,,, as 
the smallest real number so that 

Or DLs 1) 
v1 dy 
8(Z1, E)Pam <1 


Ss Bs,55 > for all a vie 


Tt 
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If X is such that 


then the interconnection of (X, X22) is also finite gain stable. 

As a special case, let X, &1, X2 represent linear systems acting on finite energy signals. Further, assume 
that stable transfer functions N, D exist where D is invertible, G, = ND~!, and N and D are normalized 
so that D? (—s)D(s) + N? (—s)N(s) = Id. Then, the class of systems in a ball with radius y > 0, measured 
in the directed gap, is given by CONE(C, R, y), where R = Id and 


_ D(s) T T 
C=Id- (ey P;(D* (—s), N° (—s)) 


where P, designates the truncation of the Laplace transform of finite energy signals to the part with poles 
in the left half plane. At the same time, ifBy,,5, < 1/y, then it can be shown that Y2 is strictly outside the 
cone ConE(C, R, y) and, therefore, stability of the interconnection of X with X2 is guaranteed for any & 
inside CoNnE(C, R, y). 

Given & and %), the computation of the directed gap reduces to a standard H..-optimization problem 
(see [1]). Also, given 1, the computation of a controller X2, which stabilizes a maximal cone around 
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2X1, reduces to a standard H.o-optimization problem [1] and forms the basis of the H..-loop shaping 
procedure for linear systems introduced in [4]. 

A second key result which prompted introducing the gap metric is the claim that the behavior of the 
feedback interconnection of & and © is “similar” to that of the interconnection of Ly and Xz if, and only 
if, the distance between © and X,, measured using the gap metric, is small (i.e., © lies within a “small 
aperture” cone around 2). The “gap” function is defined as 


8(Dy, D) = max{8(D1, Z), 8(Z, D1)} 


to “symmetrize” the distance function 3(-,-) with respect to the order of the arguments. Then, the above 
claim can be stated more precisely as follows: for each € > 0, a ¢(€) > 0 exists so that 


8(21, 2) <c(e) => Ix —xille <elldlle 


d\\ . : . . ; 
where d = ( i) is an arbitrary signal in the ambient space and x (resp. x;) represents the response 
2 


(“ +y2 
pal 
for all d and t, then 8(X, 4) <e. 


) of the feedback interconnection of (X, Xz) (resp. (1, U2)). Conversely, if ||x — x1 ||x < €||dl|x 


Defining Terms 


Ambient space: the Cartesian product space containing the inverse graph of 2 and the graph of 4. 

Distance (from a signal to a set): measured using a norm function; the infimum, over all signals in the 
set, of the norm of the difference between the signal and a signal in the set; see Equation 44.4; used 
to characterize necessary and sufficient conditions for input-output stability; see Section 44.2. 

Dynamical system: an object which produces an output signal for each input signal. 

Feedback components: the dynamical systems which make up a well-defined interconnection. 

Finite gain stable system: a dynamical system is finite gain stable if a nonnegative constant exists so that, 
for each input-output pair, the norm of the output is bounded by the norm of the input times 
the constant. 

Gain function: a function from the nonnegative real numbers to the nonnegative real numbers which is 
continuous, nondecreasing and zero when its argument is zero; used to characterize stability; 
see Section 44.2; some form of the symbol y is usually used. 

Graph (of a dynamical system): the set of ordered input-output pairs (u, y). 

Inner product: defined for signals of the same dimension defined on the semi-infinite interval; the integral 
from zero to infinity of the component-wise product of the two signals. 

Inside (or strictly outside) CONE(C, R, y): used to characterize the graph or inverse graph of a system; 
determined by whether or not signals in the graph or inverse graph satisfy certain inequalities 
involving the operators C and R and the gain function y; see Equations 44.73 and 44.74; used in 
the conic sector theorem. 

Inverse graph (of a dynamical system): the set of ordered output-input pairs (y, u). 

Norm function (|| - ||): used to measure the size of signals defined on the semi-infinite interval; examples 
are the p-norms p € [1, co] (see Equations 44.1 and 44.2). 

Parseval’s theorem: used to make connections between properties of graphs for SISO systems modeled 
with real, rational transfer functions and frequency domain characteristics of their transfer 
functions; Parseval’s theorem relates the inner product of signals to their Fourier transforms 
if they exist. For example, it states that, if two scalar signals u and y, assumed to be zero for 
negative values of time, have Fourier transforms i(jw) and y(jw) then 


(uy) = = [- #GeaGon do. 
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Passive: terminology resulting from electrical network theory; a dynamical system is passive if the inner 
product of each input-output pair is nonnegative. 

Proper separation function: a function from the nonnegative real numbers to the nonnegative real num- 
bers which is continuous, zero at zero, strictly increasing and unbounded; such functions are 
invertible on the nonnegative real numbers; used to characterize nonlinear separation theorems; 
some form of the symbol p is usually used. 

Semi-infinite interval: the time interval [0, 00). 

Signal: a “reasonable” vector-valued function defined on a finite or semi-infinite time interval; by “rea- 
sonable” we mean piecewise continuous or measurable. 

SISO systems: an abbreviation for single input, single output systems. 

Stable system: a dynamical system is stable if a gain function exists so that, for each input-output pair, 
the norm of the output is bounded by the gain function evaluated at the norm of the input. 

Strict contraction: the composition of two gain functions y; and yz is a strict contraction if a proper 
separation function p exists so that (yi + p) 0 (y2+ p) < Id, where Id(¢) = ¢ and 7 0 y2(¢) = 
yi (V¥2(¢)). Graphically, this is the equivalent to the curve (¢, yi(¢) + e(¢)) never being above the 
curve (y2(€) + p(&), &). This concept is used to state the nonlinear small gain theorem. 

Strictly passive: We have used various notions of strictly passive including input-, output-, input and 
output-, and nonlinear input and output-strictly passive. All notions strengthen the requirement 
that the inner product of the input-output pairs be positive by requiring a positive lower bound 
that depends on the 2-norm of the input and/or output. 

Truncated signal: A signal defined on the semi-infinite interval which is derived from another signal (not 
necessarily defined on the semi-infinite interval) by first appending zeros to extend the signal 
onto the semi-infinite interval and then keeping the first part of the signal and setting the rest 
of the signal to zero. Used to measure the size of finite portions of signals. 

Well-defined interconnection: An interconnection of two dynamical systems in the configuration of 
Figure 44.1 which results in another dynamical system, i.e., one in which an output signal is 
produced for each input signal. 
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Further Reading 


As mentioned at the outset, the material presented in this chapter is based on the results in [6,7,9,10]. In the 
latter, a more general feedback interconnection structure is considered where nonzero initial conditions 
can also be consider as inputs. 

Other excellent references on input-output stability include The Analysis of Feedback Systems, 1971, 
by J.C. Willems and Feedback Systems: Input-Output Properties, 1975, by C. Desoer and M. Vidyasagar. 
A nice text addressing the factorization method in linear systems control design is Control Systems 
Synthesis: A Factorization Approach, 1985, by M. Vidyasagar. A treatment of input-output stability for 
linear, infinite dimensional systems can be found in Chapter 6 of Nonlinear Systems Analysis, 1993, by 
M. Vidyasagar. That chapter also discusses many of the connections between input-output stability and 
state-space (Lyapunov) stability. Another excellent reference is Nonlinear Systems, 1992, by H. Khalil. 

There are results similar to the circle theorem that we have not discussed. They go under the heading 
of “multiplier” results and apply to feedback loops with a linear element and a memoryless, nonlinear 
element with extra restrictions such as time invariance and constrained slope. Special cases are the well- 
known Popov and off-axis circle criterion. Some of these results can be recovered using the general conic 
sector theorem although we have not taken the time to do this. Other results, like the Popov criterion, 
impose extra smoothness conditions on the external inputs which are not found in the standard problem. 
References for these problems are Hyperstability of Control Systems, 1973, V.M. Popov, the English 
translation of a book originally published in 1966, and Frequency Domain Criteria for Absolute Stability, 
1973, by K.S. Narendra and J.H. Taylor. 

Another topic closely related to these multiplier results is the structured small gain theorem for linear 
systems which lends to much of the j-synthesis control design methodology. This is described, for 
example, in -Analysis and Synthesis Toolbox, 1991, by G. Balas, J. Doyle, K. Glover, A. Packard and R. 
Smith. 

There are many advanced topics concerning input-output stability that we have not addressed. These 
include the study of small-signal stability, well-posedness of feedback loops, and control design based on 
input-output stability principles. Many articles on these topics frequently appear in control and systems 
theory journals such as IEEE Transactions on Automatic Control, Automatica, International Journal of 
Control, Systems and Control Letters, Mathematics of Control, Signals, and Systems, to name a few. 
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45.1 Introduction 


The notion of input-to-state stability (ISS) was introduced in [21]. Together with several variants, also 
discussed in this article, it provides theoretical concepts that describe various desirable stability features 
of a mapping u(-) F (-) from (time-dependent) inputs to outputs (or internal states). Prominent among 
these features are that inputs that are bounded, “eventually small,” “integrally small,” or convergent, 
should lead to outputs with the respective property. In addition, ISS and related notions quantify in 
what manner initial states affect transient behavior. The discussion in this article focuses on stability 
notions relative to globally attractive steady states, but a more general theory is also possible, that allows 
consideration of more arbitrary attractors, as well as robust and/or adaptive concepts. The reader is 
referred to the cited literature, as well as the textbooks [5,7,8,12,14,15,20], for extensions of the theory 
as well as applications, The paper [26] may also be consulted for further references and an exposition of 
many extensions of the concepts and results discussed in this chapter. 
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45.1.1 Operator and Lyapunov Stability 


Broadly speaking, there are two main competing approaches to system stability: the state-space approach 
usually associated with the name of Lyapunov, and the operator approach, of which George Zames was 
one of the main proponents and developers and which was the subject of major contributions by Sandberg, 
Willems, Safonov, and others. In the operator approach, one studies the i/o mapping: 


(x°, u(-))  yG), 
R" x [£,(0, +oo)|” > [£,4(0, +00) |? : 


that sends initial states and input functions into output functions. This includes the special case when the 
output is the internal state of a system. The notation L, refers to spaces of functions whose qth power is 
integrable; typical choices are q = 2 or q = oo. This approach permits the use of Hilbert or Banach space 
techniques, and elegantly generalizes to nonlinear systems many properties of linear systems, especially in 
the context of robustness analysis. The state-space approach, in contrast, is geared to the study of systems 
without inputs, but is better suited to the study of nonlinear dynamics, and it allows the use of geometric 
and topological ideas. The ISS notion combines these dual views of stability. 


45.1.2 The Class of Systems 


This chapter considers systems with inputs and outputs in the usual sense of control theory [24]: 


x(t) = f(x(t), u(t), y(t) = h(x(t)) 


(the arguments “t” is often omitted). There are n state variables, m input channels, and p output channels. 
States x(t) take values in Euclidean space R”, and the inputs (also called “controls” or “disturbances” 
depending on the context) are measurable locally essentially bounded maps u(-) : [0, 00) > R”. Output 
values y(t) take values in R?, The map f : R” x R” — R” is assumed to be locally Lipschitz with f (0,0) = 
0, and h: R" — R? is continuous with h(0) = 0. These two properties mean that the state x = 0 is an 
equilibrium when the input is u = 0, and the corresponding output is y = 0. Many of these assumptions 
can be weakened considerably, and the cited references should be consulted for more details. The solution, 
defined on some maximal interval [0, tmax(x°, u)), for each initial state x° and input u, is denoted as 
x(t, x°, u), and in particular, for systems with no inputs 


x(t) = f(x(t)), 


just as x(t,x°). The zero-system associated to x =f(x,u) is by definition the system with no inputs 
x = f(x, 0). Euclidean norm is written as |x|. For a function of time, typically an input or an output, ||u||, 
or ||u||.. for emphasis, is the (essential) supremum or “sup” norm (possibly +00, if u is not bounded). 
The norm of the restriction of a signal to an interval I is denoted by || u7||,o (or just||wy|l). 


45.1.3 Notions of Stability 


It is convenient to introduce “comparison functions” to quantify stability. A class Ko function is a function 
a: Ro — Rso which is continuous, strictly increasing, unbounded, and satisfies a(0) = 0 anda class KL 
function is a function B : R>9 x R>o > Rso such that B(-, t) € Koo for each t and B(r, t) decreases to zero 
as t —> 00, for each fixed r. 

For a system with no inputs x = f(x), there is a well-known notion of global asymptotic stability 
(GAS), or “0-GAS” when referring to the zero-system x = f(x, 0) associated to a given system with inputs 
x =f (x, u)) due to Lyapunov, and usually defined in “e-8” terms. It is an easy exercise to show that this 
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standard definition is in fact equivalent to the following statement: 
(ABEKL) |x(t,x°)| <B(Ix*Lt), Vx®, V =O. 
Observe that, since B decreases on ¢, in particular: 
[xCt3e)|) =P (x°1,0), Vat VES 0, 


which provides the Lyapunov-stability or “small overshoot” part of the GAS definition (because f (|x°|, 0) 
is small whenever |x°| is small, by continuity of B(-, 0) and B(0, 0) = 0), while the fact that B > 0ast — oo 
gives 


eG 21-6 (ix? 4) — 8. Vx’, 
t>0o 


which is the attractivity (convergence to steady state) part of the GAS definition. 
In [23, Proposition 7], it is shown that for each B € KL there exist two class Koo functions 01, &2 such 
that: 
B(r,t) < a2 (ai (r)e~*) , VWs,t>0, 


which means that the GAS estimate can be also written in the form: 
Ix(t,x°)| < a2 (ar(Ix*e~*) 


and thus suggests a close analogy between GAS and an exponential stability estimate |x(t, x°)| < ee | ene, 

In general, 0-GAS does not guarantee good behavior with respect to inputs. To explain why this is 
relevant, let us briefly recall the case of linear systems. A linear system in control theory is the one for 
which both f and h are linear mappings: 


x=Ax+Bu, y=Cx 


with A € R"*", Be R"*™, and C € R?*". It is well-known that a linear system is 0-GAS (or “internally 
stable”) if and only if the matrix A is a Hurwitz matrix, that is to say, all the eigenvalues of A have 
negative real parts. Such a0-GAS linear system automatically satisfies all reasonable i/o stability properties: 
bounded inputs result in bounded state trajectories as well as outputs, inputs converging to zero imply 
solutions (and outputs) converging to zero, and so forth [24]. But the 0-GAS property is not equivalent, in 
general, to i/o, or even input/state, stability of any sort. The implication that 0-GAS implies i/o stability 
is in general false for nonlinear systems. For a simple example, consider the following one-dimensional 
(n = 1) system, with scalar (m = 1) inputs: 


kext (x7 +1. 


This system is clearly 0-GAS, since it reduces to x = —x when u = 0. On the other hand, solutions 
diverge even for some inputs that converge to zero. For example, take the control u(t) = (2t + 2)~!/2 
and x° = ./2, This results in the unbounded trajectory x(t) = (2t + 2)!/2. This is in spite of the fact that 
the unforced system is GAS. Thus, the converging-input converging-state property does not hold. Even 
worse, the bounded input u = 1 results in a finite-time explosion. This example is not artificial, as it arises 


in feedback-linearization design, mentioned below. 


45.1.4 Gains for Linear Systems 
For linear systems, the three most typical ways of defining i/o stability in terms of operators 
{L?,L°} — {L7,L™} 


are as follows. The statements below should be read, more precisely, as asking that there should exist 
positive c and i such that the given estimates hold for all t > 0 and all solutions of x = Ax + Bu with 
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x(0) = x° and arbitrary inputs u(-). The estimates are written in terms of states x(t), but similar notions 
can be defined for more general outputs y = Cx. 


sa koe eae c|x(t,x°, u)| < |x°| eM sup |u(s)| 
se€[0,t] 
t 
“[? + [©”; c|x(t,x°, u)| < lems f |u(s)|* ds 
0 


t t 
oo Ge aa cf |x(s,x°, u)|° ds < [x°| +f |u(s)|? ds 
0 0 


(the missing case L© — L? is less interesting, being too restrictive). 

For linear systems, these estimates are all equivalent in the following sense: if an estimate of one type 
exists, then the other two estimates exist too, although the actual numerical values of the constants c, d 
appearing in the different estimates are not necessarily the same: they are associated to various types of 
norms on input spaces and spaces of solutions, such as “Hz” and “Ho.” gains, [4]. It is easy to see that 
existence of the above estimates is simply equivalent to the requirement that the A matrix be Hurwitz, 
that is to say, to 0-GAS, the asymptotic stability of the unforced system x = Ax. 


45.1.5 Nonlinear Coordinate Changes 


A “geometric” view of nonlinear dynamics suggests that notions of stability should be invariant under 
(nonlinear) changes of variables: under a change of variables, a system which is stable in some technical 
sense should remain stable, in the same sense, when written in the new coordinates. This principle leads 
to the ISS notion when starting from the above linear notions, as elaborated next. In this article, a change 
of coordinates is any map 
T : R°’>R" 

such that the following properties hold: T(0) = 0 (this fixes the equilibrium at x = 0), T is continuous, 
and it admits an inverse map T~!: R"->R", which is well-defined and continuous as well. In other 
words, T is a homeomorphism which fixes the origin. One could add the requirement that T should be 
differentiable, or that it be differentiable at least for x 4 0, but the discussion to follow does not require 
this additional condition. Now suppose that a system x = f(x) is exponentially stable: 


lx(t,x°)| < clx*le“’ Vt>0 (some c,r>0) 
and that a change of variables is performed: 
x(t) = T(z(t)). 
Consider, for this transformation T, the following two functions: 


a(r) := iin |T(x)| and Q(r):= ee |T(x)|, 


which are well-defined because T and its inverse are both continuous, and are both functions of class Ko 
(easy exercise). Then, 
a(|x|) < |T(x)| < a(|x|), VxeR" 
and therefore, substituting x(t, x°) = T(z(t, z°)) in the exponential stability estimate: 
a(|z(t,2°)|) < calle|e™, 
where z° = T~!(x°). Thus, the estimate in z-coordinates takes the following form: 


lz(t,z°)| < B(\z"l,t), 


where B(r, t) = a7! (ca (re~™’)) is a function of class KL. (As remarked earlier, any possible function of 
class KL can be written in this factored form, actually.) 
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In summary, the concept of GAS is rederived simply by making coordinate changes on globally 
exponentially stable systems. The same considerations, applied to systems with inputs, lead to ISS and 
related notions. In addition to the state transformation x(t) = T(z(t)), there is now also a transformation 
u(t) = S(v(t)), where S is a change of variables in the space of input values R”. Arguing analogously as 
for systems without inputs, one arrives to the following three concepts: 


L® > L® ws a (|x(t)|) < B(|x° 


»t)+ sup y(|u(s)|), 
se€[0,t] 


TPS 1? x a(|x(t)|) < B(|x° 


t 
+f y(lu(s)!) ds 
t t 
Boo [ a(x) as < an(le")+ f yduedes 
0 0 


(x(t) is written instead of the more cumbersome x(t, x°, u)). If more general outputs y = h(x) instead of 
states are the object of interest, these notions can be modified in several ways, as discussed later in the 
chapter. Unless otherwise stated, the convention when giving an estimate like the ones above is that there 
should exist comparison functions (a, a9 € Koo, B € KL) such that the estimates hold for all inputs and 
initial states. These three notions will be studied one at a time. 


45.2 ISS and Feedback Redesign 


The “L° — L®” estimate, under changes of variables, leads to the concept of ISS: there should exist some 
B € KL andy € Koo such that 


Ix(t)] < B(|x° 


st)+y (Ilulloo) (ISS) 


holds for all solutions (meaning that the estimate is valid for all inputs u(-), all initial conditions x°, 
and all tf > 0). Note that there is now no function “a” is the left-hand side because, redefining B and 
y, one can assume, without loss of generality, that « is the identity: if a(r) < B(s, t)+ y(t) holds, then 
also r < a! (B(s, t) + y(t) < a7! (2B(s, £)) + a7! (2y(t)); since a7! (2B(.,-)) € KL and a7! (2y(-)) € Koos 
an estimate of the same type, but now with no “a,” is obtained. In addition, note that the supremum 
SUP 510.4] y(|u(s)|) over the interval [0, t] is the same as y(||u0,¢]lloo) = Y(SUP.<r0,4(1U(s)I)), because the 
function y is increasing, so one may replace this term by y(||u||..), where ||u||,9 = SUP s¢[0,00) y(ju(s)|) 
is the sup norm of the input, because the solution x(t) depends only on values u(s),s < t (so, one could 
equally well consider the input that has values = 0 for all s > ft). 

Note that a potentially weaker definition might be that the ISS estimate should hold merely for all 
t € [0, tmax(x°, u)). However, this potentially different definition turns out to be equivalent. Indeed, if 
the estimate holds a priori only on such a maximal interval of definition, then, since the right-hand 
is bounded on [0, T], for any T > 0 (recall that inputs are by definition assumed to be bounded on 
any bounded interval), it follows that the maximal solution of x(t, x°, uv) is bounded, and therefore that 
tmax(x°, u) = +00 (e.g., Proposition C.3.6 in [24]). In other words, the ISS estimate holds for all t > 0 
automatically, if it is required to hold merely for maximal solutions. 

Since, in general, max{a, b} < a+b < max{2a, 2b}, one can restate the ISS condition in a slightly 
different manner, namely, asking for the existence of some B € KL and y € Koo (in general, different 
from the ones in the ISS definition) such that 


Ix()| < max {B(|x"l,£). y (Ilullloo) } 


holds for all solutions. Such redefinitions, using “max” instead of sum, are also possible for each of the 
other concepts to be introduced later. 
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~ |x| 


{= tal. 


> 


FIGURE 45.1 ISS combines overshoot and asymptotic behavior. 


Intuitively, the definition of ISS requires that, for t large, the size of the state must be bounded by some 
function of the sup norm—that is to say, the amplitude—of inputs (because B(|x° ,t) > Oast > o0).On 
the other hand, the B(|x° ,0) term may dominate for small t, and this serves to quantify the magnitude 
of the transient (overshoot) behavior as a function of the size of the initial state x° (Figure 45.1). 

The ISS superposition theorem, discussed later, shows that ISS is, in a precise mathematical sense, the 
conjunction of two properties, one of them dealing with asymptotic bounds on |x°| as a function of the 
magnitude of the input, and the other one providing a transient term obtained when one ignores inputs. 


45.2.1 Linear Case, for Comparison 
For internally stable linear systems * = Ax + Bu, the variation of parameters formula gives immediately 
the following inequality: 
Il < BCA) |x°| +y llulloo s 
where 
[e,@) 
B(t) = |e’ | > 0 and y=|Bll i: | es! | ds < oo. 
0 


This is a particular case of the ISS estimate, |x(t)| < B(|x°|,t) +y (Iltll oo) with linear comparison func- 
tions. 


45.2.2 Feedback Redesign 


The notion of ISS arose originally as a way to precisely formulate, and then answer the following question. 
Suppose that, as in many problems in control theory, a system x = f(x, u) has been stabilized by means of 
a feedback law u = k(x) (Figure 45.2), that is to say, k was chosen such that the origin of the closed-loop 
system x = f(x,k(x)) is globally asymptotically stable. (See e.g. [25] for a discussion of mathematical 


>| x = f(x, v) 


u = k(x) —<<—=— 


FIGURE 45.2 Feedback stabilization, closed-loop system x = f (x, k(x)). 


Input-to-State Stability 45-7 


i x= f(x, v) 


u = k(x) 


FIGURE 45.3 Actuator disturbances, closed-loop system x = f(x, k(x) + d). 


aspects of state feedback stabilization.) Typically, the design of k was performed by ignoring the effect of 
possible input disturbances d(-) (also called actuator disturbances). These “disturbances” might represent 
true noise or perhaps errors in the calculation of the value k(x) by a physical controller, or modeling 
uncertainty in the controller or the system itself. What is the effect of considering disturbances? In order to 
analyze the problem, d is incorporated into the model, and one studies the new system x = f(x, k(x) + d), 
where d is seen as an input (Figure 45.3). One may then ask what is the effect of d on the behavior of the 
system. 

Disturbances d may well destabilize the system, and the problem may arise even when using a routine 
technique for control design, feedback linearization. To appreciate this issue, take the following very 
simple example. Given is the system 


K =f (x,u) =x+(x? + 1)u. 


In order to stabilize it, substitute u = i7/(x? + 1) (a preliminary feedback transformation), rendering the 


system linear with respect to the new input u: x = x+ 4, and then use u = —2x in order to obtain the 
closed-loop system x = —x. In other words, in terms of the original input u, the feedback law is 
—2x 
k(x) = Day? 
x +1 


so that f(x, k(x)) = —x. This is a GAS system. The effect of the disturbance input d is analyzed as follows. 
The system x = f(x, k(x) +d) is 
k= —x + (x* +1)d. 


As seen before, this system has solutions that diverge to infinity even for inputs d that converge to 
zero; moreover, the constant input d= 1 results in solutions that explode in finite time. Thus k(x) = 
—2x/(x” + 1) was not a good feedback law, in the sense that its performance degraded drastically once 
actuator disturbances were taken into account. 

The key observation for what follows is that, if one adds a correction term “ 
for k(x), so that now 


‘“_x” to the above formula 
—2x 
~ x24] 


k(x) 


x; 
then the system x = f(x, k(x) + d) with disturbance d as input becomes, instead 
k= 2x —x? +(x? + 1)d 


and this system is much better behaved: it is still GAS when there are no disturbances (it reduces to x = 
—2x — x?) but, in addition, it is ISS (easy to verify directly, or appealing to some of the characterizations 
mentioned later). Intuitively, for large x, the term —x? serves to dominate the term (x? + 1)d, for all 
bounded disturbances d(-), and this prevents the state from getting too large. 
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d oy > x= f(x u) 


FIGURE 45.4 Different feedback ISS-stabilizes. 


45.2.3 A Feedback Redesign Theorem for Actuator Disturbances 


This example is an instance of a general result, which says that whenever there is some feedback law that 
stabilizes a system, there is also a (possibly different) feedback so that the system with external input d 
(Figure 45.4) is ISS. 


Theorem 45.1: [21] 


Consider a system affine in controls 


m 


& =f(x,u) =go(x) + >> uigi(x) (go(0) = 0) 


i=1 


and suppose that there is some differentiable feedback law u = k(x) so that 
i = f(x, k(x)) 
has x = 0 as a GAS equilibrium. Then, there is a feedback law u = k(x) such that 
= f(x, k(x) +d) 


is ISS with input d(-) 


The proof is very easy, once the appropriate technical machinery has been introduced: one starts by 
considering a smooth Lyapunov function V for GAS of the origin in the system « = f(x, k(x)) (such a 
V always exists, by classical converse theorems); then k(x) := —(LgV(x))? = —(VV(x)G(x))?, where G 
is the matrix function whose columns are the gj,i=1,...,m and T indicates transpose, provides the 
necessary correction term to add to k. This term has the same degree of smoothness as the vector fields 
making up the original system. Somewhat less than differentiability of the original k is enough for this 
argument: continuity is enough. However, if no continuous feedback stabilizer exists, then no smooth 
V can be found. (Continuous stabilization of nonlinear systems is basically equivalent to the existence 
of what are called smooth control-Lyapunov functions, see e.g. [25].) In that case, if only discontinuous 
stabilizers are available, the result can still be generalized, see [17], but the situation is harder to analyze, 
since even the notion of “solution” of the closed-loop system x = f (x, k(x)) has to be carefully defined. 

There is also a redefinition procedure for systems that are not affine on inputs, but the result as stated 
above is false in that generality, and is much less interesting; see [22] for a discussion. 

The above feedback redesign theorem is merely the beginning of the story. The reader is referred to 
the book [15], and the references given later, for many further developments on the subjects of recursive 
feedback design, the “backstepping” approach, and other far-reaching extensions. 
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45.3 Equivalences for ISS 


This section reviews results that show that ISS is equivalent to several other notions, including asymptotic 
gain (AG), existence of robustness margins, dissipativity, and an energy-like stability estimate. 


45.3.1 Nonlinear Superposition Principle 


Clearly, if a system is ISS, then the system with no inputs x = f(x, 0) is GAS: the term ||u||,. vanishes, 
leaving precisely the GAS property. In particular, then, the system x = f(x, u) is 0-stable, meaning that 
the origin of the system without inputs x = f(x, 0) is stable in the sense of Lyapunov: for each ¢ > 0, there 
is some 8 > 0 such that |x°| < 8 implies |x(t, x) < e. (In comparison function language, one can restate 
0-stability as: there is some y € K such that |x(¢, co < y(|x° ») holds for all small x°.) 

On the other hand, since B(|x° ,t) > 0 as t > o, for t large one has that the first term in the ISS 
estimate |x(t)| < max {B(1x°L, t),y (IIx4ll 00) } vanishes. Thus, an ISS system satisfies the following “AG” 
property: there is some y € Koo so that: 


Jim, |x(t,x°, w)| < y(Ilullloo)s Vx, uC) (AG) 


(see Figure 45.5). In words, for all large enough tf, the trajectory exists, and it gets arbitrarily close to 
a sphere whose radius is proportional, in a possibly nonlinear way quantified by the function y, to 
the amplitude of the input. In the language of robust control, the estimate (AG) would be called an 
“altimate boundedness” condition; it is a generalization of attractivity (all trajectories converge to zero, 
for a system x = f(x) with no inputs) to the case of systems with inputs; the “lim sup” is required 
since the limit of x(t) as t + oo may well not exist. From now on (and analogously when defining 
other properties), we will just say “the system is AG” instead of the more cumbersome “satisfies the 
AG property.” 

Observe that, since only large values of t matter in the limsup, one can equally well consider merely 
tails of the input u when computing its sup norm. In other words, one may replace y(||ull.9) by 
y(limy-s +00 |u(£)|), or (since y is increasing), lim) +ooy(|u(t)]). 


x(0) 


FIGURE 45.5 Asymptotic gain property. 
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The surprising fact is that these two necessary conditions are also sufficient. This is summarized by the 
ISS superposition theorem: 


Theorem 45.2: [29] 


A system is ISS if and only if it is 0-stable and AG. 


The basic difficulty in the proof of this theorem is in establishing uniform convergence estimates for 
the states, that is, in constructing the 6 function in the ISS estimate, independently of the particular input. 
As in optimal control theory, one would like to appeal to compactness arguments (using weak topologies 
on inputs), but there is no convexity to allow this. The proof hinges upon a lemma given in [29], which 
may be interpreted [6] as a relaxation theorem for differential inclusions, relating GAS of an inclusion 
x € F(x) to GAS of its convexification. 

A minor variation of the above superposition theorem is as follows. Let us consider the limit property 
(LIM): 

inf [x(t,x°, u)| < y(Ilulloo)» Vx", uC) (LIM) 


(for some y € Ko). 


Theorem 45.3: [29] 


A system is ISS if and only if it is 0-stable and LIM. 


45.3.2 Robust Stability 


In this article, a system is said to be robustly stable if it admits a margin of stability p, that is, a smooth 
function p € Koo so the system 


x = g(x, d) := f(x, dp(|x|)) 


is GAS uniformly in this sense: for some B € KL, 


|x(t, x°, d)| < B(|x° 


,t) 


for all possible d(-) :[0,00) + [—1,1]”. An alternative way to interpret this concept (cf. [28]) is as 
uniform GAS of the origin with respect to all possible time-varying feedback laws A bounded by p: 
|A(t, x)| < p(|x|). In other words, the system 


x =f (x, A(t, x)) 


(Figure 45.6) is stable uniformly over all such perturbations A. In contrast to the ISS definition, which 
deals with all possible “open-loop” inputs, the present notion of robust stability asks about all possible 
closed-loop interconnections. One may think of A as representing uncertainty in the dynamics of the 
original system, for example. 


Theorem 45.4: [28] 


A system is ISS if and only if it is robustly stable. 
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— # = f(x, u) 


FIGURE 45.6 Margin of robustness. 


Intuitively, the ISS estimate |x(t)| < max {B(Ix°L, t),y (II u||l ea) says that the B term dominates as long 
as |u(t)| < |x(t)| for all t, but |u(t)| « |x(t)| amounts to u(t) = d(t).e(|x(t)|) with an appropriate function 
p. This is an instance of a “small gain” argument, see below. One analog for linear systems is as follows: if 
A is a Hurwitz matrix, then A + Q is also Hurwitz, for all small enough perturbations Q; note that when 
Q is a nonsingular matrix, |Qx| is a Koo function of |x|. 


45.3.3 Dissipation 


Another characterization of ISS is as a dissipation notion stated in terms of a Lyapunov-like function. A 
continuous function V : R” —> R is said to be a storage function if it is positive definite, that is, V(0) = 0 
and V(x) > 0 for x #0, and proper, that is, V(x) — 00 as |x| — oo. This last property is equivalent to 
the requirement that the sets V—!({0,.A]) should be compact subsets of IR”, for each A > 0, and in the 
engineering literature it is usual to call such functions radially unbounded. It is an easy exercise to show 
that V : R” > R isa storage function if and only if there exist a, @ € Koo such that 


a(|x]) < V(x) < a(|x|) Vx eR” 


(the lower bound amounts to properness and V(x) > 0 for x £0, while the upper bound guarantees 
V(0) = 0). For convenience, V : R” x R” — R is the function 


V(x, u) = VV(x) - f(x, u), 


which provides, when evaluated at (x(t), u(t)), the derivative dV (x(t))/dt along solutions of x = f(x, u). 
An ISS-Lyapunov function for x = f(x, u) is by definition a smooth storage function V for which there 
exist functions y, @ € Koo so that 


Vix,u) < —a(|x|)+y(lul), Vx, u. (L-ISS) 


Integrating, an equivalent statement is that, along all trajectories of the system, there holds the following 


dissipation inequality: 
h 


V(x(t2)) — Vx(t)) <| w(u(s), x(s)) ds, 


ty 
where, using the terminology of [31], the “supply” function is w(u, x) = y(|u|) — a(|x|). For systems with 
no inputs, an ISS-Lyapunov function is precisely the same object as a Lyapunov function in the usual 
sense. 


Theorem 45.5: [28] 


A system is ISS if and only if it admits a smooth ISS-Lyapunov function. 
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Since —a(|x|) < —a(a@~1(V(x))), the ISS-Lyapunov condition can be restated as 


for some @ € Koo. In fact, one may strengthen this a bit [19]: for any ISS system, there is a always a smooth 
ISS-Lyapunov function satisfying the “exponential” estimate V(x, u) < —V(x) + y(\u)). 

The sufficiency of the ISS-Lyapunov condition is easy to show, and was already in the original 
paper [21]. A sketch of the proof is as follows, assuming for simplicity a dissipation estimate in the 
form V(x, uv) < —a(V(x)) + y(|u|). Given any x and u, either a(V(x)) < 2y(|ul) or V< —a(V)/2. From 
here, one deduces by a comparison theorem that, along all solutions, 


V(x(t)) < max {B(V(x°), 1), a! (2y(Ilulloo))} 5 


where the KL function f(s, t) is the solution y(t) of the initial value problem 


1 
p=—say)+yu), yO) =s. 


Finally, an ISS estimate is obtained from V(x°) < a(x°). 

The proof of the converse part of the theorem is based upon first showing that ISS implies robust stability 
in the sense already discussed, and then obtaining a converse Lyapunov theorem for robust stability for the 
system x = f(x, dp(|x|)) = g(x, d), which is asymptotically stable uniformly on all Lebesgue-measurable 
functions d(-) : R>9 — B(0, 1). This last theorem was given in [16], and is basically a theorem on Lyapunov 
functions for differential inclusions. The classical result of Massera [18] for differential equations (with 
no inputs) becomes a special case. 


45.3.4 Using “Energy” Estimates Instead of Amplitudes 


In linear control theory, Hoo theory studies L? > L?-induced norms, which under coordinate changes 
leads to the following type of estimate: 


t 


t 
i a(|x(s)]) ds < aotlstd+ f y(lu(s)]) ds 
0 0 


along all solutions, and for some a, ao, y € Koo. Just for the statement of the next result, a system is said 
to satisfy an integral-integral estimate if for every initial state x° and input u, the solution x(t, x°, u) is 
defined for all t > 0 and an estimate as above holds. (In contrast to ISS, this definition explicitly demands 
that tmax = ©.) 


Theorem 45.6: [23] 


A system is ISS if and only if it satisfies an integral-integral estimate. 


This theorem is quite easy to prove, in view of previous results. A sketch of the proof is as follows. If the 
system is ISS, then there is an ISS-Lyapunov function satisfying V(x, u) < —V(x) + y(|ul), so, integrating 
along any solution: 


t 


t t 
/ V(x(s)) ds <| V(x(s)) ds + V(x(t)) < vixcoy +f y(u(s)|) ds 
0 0 0 


and thus an integral-integral estimate holds. Conversely, if such an estimate holds, one can prove that 
x = f(x, 0) is stable and that an AG exists. 
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45.4 Cascade Interconnections 


One of the main features of the ISS property is that it behaves well under composition: a cascade 
(Figure 45.7) of ISS systems is again ISS, see [21]. This section sketches how the cascade result can 
also be seen as a consequence of the dissipation characterization of ISS, and how this suggests a more 
general feedback result. For more details regarding the rich theory of ISS small-gain theorems, and their 
use in nonlinear feedback design, the references should be consulted. Consider a cascade as follows: 


zZ=f(z,x), 
x = g(x, u), 


where each of the two subsystems is assumed to be ISS. Each system admits an ISS-Lyapunov function 
V;. But, moreover, it is always possible (see [27]) to redefine the V;’s so that the comparison functions for 
both are matched in the following way: 


Vi(z, x) < O(\x|) — a(|zI), 
Vo(x, u) < A(\ul) — 20(|xl). 


Now it is obvious why the full system is ISS: simply use V := V; + V2 as an ISS-Lyapunov function for 
the cascade: 
V((x,z),u) < O(lul) — O(|x|) — a(|zl). 


Of course, in the special case in which the x-subsystem has no inputs, this also proved that the cascade 
of a GAS and an ISS system is GAS. 
More generally, one may allow a “small gain” feedback as well (Figure 45.8). That is, one allows 
inputs u = k(z) as long as they are small enough: 
Ik(z)| < 0 (1 —e)a(lzl)). 
The claim is that the closed-loop system 
z=f(z,x), 
x = g(x, k(x)) 
is GAS. This follows because the same V is a Lyapunov function for the closed-loop system; for (x, z) # 0: 
A(ul) <1 —e)a(lzl) + V(x,z) < —O(|x|) — ea(lzl) <0. 


A far more interesting version of this result, resulting in a composite system with inputs being itself 
ISS, is the ISS small-gain theorem due to Jiang et al. [10]. 


u 
x Zz 


FIGURE 45.7 Cascade. 


FIGURE 45.8 Adding a feedback to the cascade. 
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45.5 Integral ISS 


Several different properties, including “integral-to-integral” stability, dissipation, robust stability margins, 
and AG properties, were all shown to be exactly equivalent to ISS. Thus, it would appear to be difficult 
to find a general and interesting concept of nonlinear stability that is truly distinct from ISS. One such 
concept, however, does arise when considering a mixed notion which combines the “energy” of the input 
with the amplitude of the state. It is obtained from the “L? > L®©” gain estimate, under coordinate 
changes, and it provides a genuinely new concept [23]. 

A system is said to be integral-input-to-state stable (iISS) provided that there exist a, y € Koo and 
6 € KL such that the estimate 


t 
a (|x(t)|) < B(Ix*|, t) + / y(u(s)]) ds (iISS) 
0 


holds along all solutions. Just as with ISS, one could state this property merely for all times f € tmax(x°, u). 
Since the right-hand side is bounded on each interval [0, t] (because, inputs are by definition assumed 
to be bounded on each finite interval), it is automatically true that tmax(x°, u) = +00 if such an estimate 
holds along maximal solutions. So forward-completeness (solution exists for all t > 0) can be assumed 
with no loss of generality. 


45.5.1 Other Mixed Notions 


A change of variables starting from a system that satisfies a finite operator gain condition from L? to L4, 
with p # q both finite, leads naturally to the following type of “weak integral-to-integral” mixed estimate: 


t t 
i, a(|x(s)]) ds < (|x|) ba (/ (0 ds) 
0 0 


«>> 


for appropriate K.o functions (note the additional “w”). See [3] for more discussion on how this estimate 
is reached, as well as the following result: 


Theorem 45.7: [3] 
A system satisfies a weak integral-to-integral estimate if and only if it is iISS. 
Another interesting variant is found when considering mixed integral/supremum estimates: 
t 
a(lx(01 = Bla’ bt) + f° yall) ds-+ yaClilo) 
0 


for suitable B € KL and a, yj € Koo. One then has: 


Theorem 45.8: [3] 


A system satisfies a mixed estimate if and only if it is iISS. 


45.5.2 Dissipation Characterization of iISS 


Recall that a storage function is a continuous V : R” — R which is positive definite and proper. A smooth 
storage function V is an i[SS-Lyapunov function for the system x = f(x, u) if there are a y € Koo and an 
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a: [0,-+oo) — [0, +00) which is merely positive definite (i.e., a(0) = 0 and a(r) > 0 for r > 0) such that 
the inequality 
V(x, u) < —a(|x|) + y(lul) (L-iISS) 


holds for all (x, uv) € R” x R”. To compare, recall that an ISS-Lyapunov function is required to satisfy an 
estimate of the same form but where « is required to be of class Koo; since every Koo function is positive 
definite, an ISS-Lyapunov function is also an ilSS-Lyapunov function. 


Theorem 45.9: [2] 


A system is iISS if and only if it admits a smooth iISS-Lyapunov function. 


Since an ISS-Lyapunov function is also an iISS one, ISS implies iISS. However, iISS is a strictly weaker 
property than ISS, because a may be bounded in the ilSS-Lyapunov estimate, which means that V may 
increase, and the state become unbounded, even under bounded inputs, so long as y(|u(f)|) is larger than 
the range of a. This is also clear from the iISS definition, since a constant input with |u(t)| =r results in 
a term in the right-hand side that grows like rt. 

An interesting general class of examples is given by bilinear systems 


m 
x= (44 Soa) hm 


i=1 


for which the matrix A is Hurwitz. Such systems are always iISS (see [23]), but they are not in general 
ISS. For instance, in the case when B = 0, boundedness of trajectories for all constant inputs already 
implies that A + )7", u;A; must have all eigenvalues with nonpositive real part, for all u € R™, which is 
a condition involving the matrices A; (e.g., x = —x + ux is iISS but it is not ISS). 

The notion of iISS is useful in situations where an appropriate notion of detectability can be verified 
using LaSalle-type arguments. There follow two examples of theorems along these lines. 


Theorem 45.10: [2] 


A system is iISS if and only if it is 0-GAS and there is a smooth storage function V such that, for some 
GEKQ: 
V(x, u) < o(|ul) 


for all (x, u). 


The sufficiency part of this result follows from the observation that the 0-GAS property by itself already 
implies the existence of a smooth and positive-definite, but not necessarily proper, function Vo such that 
Vo < Yo(|ul) — ao(|x]) for all (x, u), for some yo € Koo and positive-definite a9 (if Vo were proper, then it 
would be an ilSS-Lyapunov function). Now one uses Vo + V as an iISS-Lyapunov function (V provides 
properness). 


Theorem 45.11: [2] 


A system is iISS if and only if there exists an output function y = h(x) (continuous and with h(0) = 
0), which provides zero-detectability (u=0 and y=0= x(t) > 0) and dissipativity in the following 


45-16 Control System Advanced Methods 


sense: there exists a storage function V and o € Koo, a positive definite, so that: 
V(x, u) < o(|ul) — a(h(x)) 
holds for all (x, u). 


The paper [3] contains several additional characterizations of iISS. 


45.5.3 Superposition Principles for iISS 


There are also AG characterizations for iISS. A system is bounded energy weakly converging state (BEWCS) 
if there exists some o € Koo so that the following implication holds: 


+00 
/ o(|u(s)|)ds <<+oo => liminf |x(t,x°, u)| =0 (BEWCS) 
0 t+ +00 


(more precisely: if the integral is finite, then tmax(x°, u) = +-oo and the lim infis zero). It is bounded energy 
frequently bounded state (BEFBS) if there exists some o € Koo so that the following implication holds: 


+00 
/ o(|u(s)|)ds <+o0o => lim inf |x(t, x°, u)| < +00 (BEFBS) 
0 —>+00 


(again, meaning that tmax(x°, uw) = +oo and the lim inf is finite). 


Theorem 45.12: [1] 


The following three properties are equivalent for any given system x = f(x, u): 


¢ The system is iISS 
¢ The system is BEWCS and 0-stable 
¢ The system is BEFBS and 0-GAS 


45.6 Output Notions 


Until now, the chapter discussed only stability of states with respect to inputs. For systems with outputs 
x =f (x, u), y = h(x), several new notions can be introduced. 


45.6.1 Input-to-Output Stability 


If one simply replaces states by outputs in the left-hand side of the estimate defining ISS, there results the 
notion of input-to-output stability (IOS): there exist some B € KL and y € Koo such that 


y(t)| < BUx°L t) +Y (lulloo) (IOS) 


holds for all solutions, where y(t) = h(x(t, x°, u)). (Meaning that the estimate is valid for all inputs u/(-), all 
initial conditions x°, and all t > 0, and imposing as a requirement that the system be forward complete, 
that is, tmax(x°, u) = oo for all initial states x° and inputs u.) As earlier, x(t), and hence y(t) = h(x(t)), 
depend only on past inputs (“causality”), so one could have used just as well simply used the supremum 
of |u(s)| for s > t in the estimate. 

A system is bounded-input bounded-state stable (BIBS) if, for some o € Ko, the following estimate 


|x(t)| < max {o(|x"|), o(||Ulloo)} 


holds along all solutions. (Note that forward completeness is a consequence of this inequality, even if it is 
only required on maximal intervals, since the state is upper bounded by the right-hand side expression.) 
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An IOS-Lyapunov function is any smooth function V : R” — Rso so that, for some aj € Koo: 


ay (|h(x)|) < V(x) <ap(Ix|), VxeR", we R™ 


and, for all x, u: 
V(x) > a3(|ul) > VV(x) f(x, u) < 0. 


One of the key results for IOS is as follows: 


Theorem 45.13: [30] 


A BIBS system is IOS if and only if it admits an IOS-Lyapunov function. 


45.6.2 Detectability and Observability 


Recall (see [24] for precise definitions) that an observer for a given system with inputs and outputs 
x =f (x, u), y = h(x) is another system which, using only information provided by past input and output 
signals, provides an asymptotic (i.e., valid as t > 00) estimate X(t) of the state x(t) of the system of interest 
(Figure 45.9). 

One may think of the observer as a physical system or as an algorithm implemented by a digital 
computer. The general problem of building observers is closely related to “incremental” ISS-like notions, 
a subject not yet studied enough. This chapter will limit itself to an associated but easier question. When 
the ultimate goal is that of stabilization to an equilibrium, say x = 0 in Euclidean space, sometimes a 
weaker type of estimate suffices: it may be enough to obtain a norm-estimator which provides merely an 
upper bound on the norm |x(t)| of the state x(t); see [9,11,19]. 

Suppose that an observer exists for a given system. Since x° = 0 is an equilibrium for « = f(x, 0), and 
also h(0) = 0, the solution x(t) = 0 is consistent with u =0 and y = 0. Thus, the estimation property 
x(t) — x(t) > 0 implies that x(t) + 0. Now consider any state x° for which u=0 and y = 0, that is, 
so that h(x(t,x°,0)) = 0. The observer output, which can only depend on u and y, must be the same x 
as when x° = 0, so x(t) — 0; then, using once again the definition of observer X(t) — x(t, x°, 0) > 0, it 
follows that x(t, x°,0) — 0. In summary, a necessary condition for the existence of an observer is that the 
“subsystem” of x = f(x, u), y = h(x), consisting of those states for which u = 0 produces the output y = 0, 
must have x = 0 as a GAS state (Figure 45.10); one says in that case that the system is zero detectable. 
(For linear systems, zero detectability is equivalent to detectability or “asymptotic observability” [24]: 
two trajectories than produce the same output must approach each other. But this equivalence need not 
hold for nonlinear systems.) In a nonlinear context, zero detectability is not “well-posed” enough: to 
get a well-behaved notion, one should add explicit requirements to ask that small inputs and outputs 
imply that internal states are small too (Figure 45.11), and that inputs and outputs converging to zero 


FIGURE 45.9 Observer provides estimate x of state x; x(t) — x(t) > 0as t > oo. 
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FIGURE 45.10 Zero detectability. 
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———>| => x= 0 =e 


FIGURE 45.11 Small inputs and outputs imply small states. 


u—>0———>| x30 aaa) 


FIGURE 45.12 Converging inputs and outputs imply convergent states. 


as t —> oo implies that states do, too (Figure 45.12). These properties are needed so that “small” errors 
in measurements of inputs and outputs processed by the observer give rise to small errors. Furthermore, 
one should impose asymptotic bounds on states as a function of input/output bounds, and it is desirable 
to quantify “overshoot” (transient behavior). This leads us to the following notion. 


45.6.3 Dualizing ISS to OSS and IOSS 


A system is input/output to state stable (IOSS) if, for some B € KL and yy, Vy € Koos 


x(t) < B(Ix°|,t) + y1 (luo. i) +y2 ([lyto.| aa) (IOSS) 


for all initial states and inputs, and all t € [0, T:,,,). Just as ISS is stronger than 0-GAS, IOSS is stronger 
than zero detectability. A special case is when one has no inputs, output to state stability: 


Ix(t,x°)] < B(x) + ((lylpoa |.) 


and this is formally “dual” to ISS, simply replacing inputs u by outputs in the ISS definition. This duality 
is only superficial, however, as there seems to be no useful way to obtain theorems for OSS by dualizing 
ISS results. (Note that the outputs y depend on the state, not vice versa.) 


45.6.4 Lyapunov-Like Characterization of IOSS 


To formulate a dissipation characterization, we define an IOSS-Lyapunov function as a smooth storage 
function so that 
VV (x) f(x, u) < —a1 (x1) + a2 (ul) + 03 (Ly1) 


for all x € R", ue R™,y € R?. The main result is 


Theorem 45.14: [13] 


A system is IOSS if and only if it admits an IOSS-Lyapunov function. 


45.6.5 Norm-Estimators 
A state-norm-estimator (or state-norm-observer) for a given system is another system 
Z=g(z,u,y), with output k: R’x RP > Rso 


evolving in some Euclidean space R°, and driven by the inputs and outputs of the original system. It is 
required that the output k should be IOS with respect to the inputs u and y, and the true state should be 
asymptotically bounded in norm by some function of the norm of the estimator output, with a transient 
(overshoot) which depends on both initial states. Formally 


Input-to-State Stability 45-19 


+ there are 1,2 € K and € KCL so that, for each initial state z° € R’, and inputs u and y, and every 
t in the interval of definition of the solution z(-, z°, u, y): 


k (z(t,z°, uy) y(t) < (iz) +91 (llulron}) + %2 (lylro.4| 


); 


¢ there are p €K,B € KCL so that, for all initial states x° and z° of the system and observer, and every 
input u: 
|x(t, x’, u)| < B([x°]+ [2°], +0 (k (z(t, 2, u, Yx9u)> ¥x0,u(t))) 


for all t € [0, tmax(x°, u)), where y,o 4 (t) = y(t, x°, u). 


Theorem 45.15: [13] 


A system admits a state-norm-estimator if and only if it is IOSS. 


45.7 The Fundamental Relationship among ISS, IOS, and IOSS 


The definitions of the basic ISS-like concepts are consistent and related in an elegant conceptual manner, 
as follows: 

A system is ISS if and only if it is both IOS and IOSS. 

In informal terms: 


External stability and detectability <> Internal stability 


as it is the case for linear systems. Intuitively, consider the three possible signals in Figure 45.13 

The basic idea of the proof is as follows. Suppose that external stability and detectability hold, and 
take an input so that u — 0. Then y > 0 (by external stability), and this then implies that x + 0 (by 
detectability). Conversely, if the system is internally stable, then it is i/o stable and detectable. Suppose 
that u — 0. By internal stability, x — 0, and this gives y(t) — 0 (i/o stability). Detectability is even easier: 
if both u(t) > 0 and y(t) > 0, then in particular u — 0, so x — 0 by internal stability. The proof that ISS 
is equivalent to the conjunction of IOS and IOSS must keep careful track of the estimates, but the idea is 
similar. 


45.8 Response to Constant and Periodic Inputs 


Systems x = f(x, u) that are ISS have certain noteworthy properties when subject to constant or, more 
generally periodic, inputs. 

Let V be an ISS-Lyapunov function that satisfies the inequality V(x, u) < —V(x) + y({ul) for all x, u, 
for some y € Koo. To start with, suppose that u is any fixed bounded input, and let a := y(||ul|,.), pick any 
initial state x°, and consider the solution x(t) = x(t, x°, z) for this input. Letting v(t) := V(x(¢)), it follows 
that v(t) + v(t) < aso, using e! as an integrating factor, v(t) < a+ e ‘(v(0) — a) forall t > 0. In particular, 
if v(0) <a it will follow that v(t) <a for all t > 0, that is to say, the sublevel set K := {x | V(x) <a} 
is a forward-invariant set for this input: if x° € K then x(t) =x(t,x°,u) € K for all t > 0. Therefore, 


u 30 ———> x30 |W» y>0 


FIGURE 45.13 Convergent input, state, and/or output. 
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Mr: x°+> x(T,x°, iu) is a continuous mapping from K into K, for each fixed T > 0, and thus, provided 
that K has a fixed-point property (every continuous map M : K — K has some fixed point), then for each 
T > 0 there exists some state x° such that x(T, x°, u) = x°. The set K indeed has the fixed-point property, 
as does any sublevel set of a Lyapunov function. To see this, note that V is a Lyapunov function for the 
zero-input system x = f(x, 0), and thus, if B is any ball which includes K in its interior, then the map 
Q: B-> K which sends any & € B into x(tg, &), where fg is the first time such that x(t, &) € K is continuous 
(because the vector field is transversal to the boundary of K since V V(x).f(x,0) < 0), and is the identity 
on K (that is, Q is a topological retraction). A fixed point of the composition Mo Q: B—> B is a fixed 
point of M. 

Now suppose that i is periodic of period T, u(t + T) = u(t) for all t > 0, and pick any x° which is a 
fixed point for Mr. Then the solution x(t, x°, z) is periodic of period T as well. In other words, for each 
periodic input, there is a solution of the same period. In particular, if u is constant, one may pick for each 
h> 0a state x; so that x(h, x}, u) = xp, and therefore, picking a convergent subsequence x;, > x gives 
that 0 = (1/h)(x(h, xp, u) — xp,) > f(x, 4), so f(x, u) = 0. Thus one also has the conclusion that for each 
constant input, there is a steady state. 
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46.1 The Problem of Feedback Linearization 


A basic problem in control theory is how to use feedback in order to modify the original internal dynamics 
ofa controlled plant so as to achieve some prescribed behavior. In particular, feedback may be used for the 
purpose of imposing, on the associated closed-loop system, the (unforced) behavior of some prescribed 
autonomous linear system. When the plant is modeled as a linear time-invariant system, this is known as 
the problem of pole placement, while, in the more general case of a nonlinear model, this is known as the 
problem of feedback linearization (see [1-4]). 

The purpose of this chapter is to present some of the basic features of the theory of feedback 
linearization. 

Consider a plant modeled by nonlinear differential equations of the form 


x=f(x)+g(x)u (46.1) 
y = h(x) (46.2) 


having internal state x = (x,,x2,--- ,X,) € R", control input u€ R and measured output y € R. The 
functions 


Fi (x1, X25 °° Xn) 


fw= fo(x1,x2,°++ Xn) 


(X15 X25°°° Xn) 
S1(X15X25°°° Xn) 
2(X1,X25°+* s Xn) 
gla) = | Beery, 
Sn(X1,X2,°°° en) 


h(x) = h(x1, x2,+++ » Xn) 


46-1 
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are nonlinear functions of their arguments that are assumed to be differentiable a sufficient number 
of times. 

Changes in the description and in the behavior of this system will be investigated under two types of 
transformations: (1) changes of coordinates in the state space and (2) static state feedback control laws, 
ie., memoryless state feedback laws. 

In the case of a linear system, 


x= Ax+ Bu (46.3) 
y=Cx (46.4) 


a static state feedback control law takes the form 
u= Fx + Gv, (46.5) 


in which v represents a new control input and F and G are matrices of appropriate dimensions. Moreover, 
only linear changes of coordinates are usually considered. This corresponds to the substitution of the 
original state vector x with a new vector z related to x by a transformation of the form 


zZ=Tx 


where T is a nonsingular matrix. Accordingly, the original description of the system of Equations 46.3 
and 46.4 is replaced by a new description 


z—Az+Bu 
y=Cz 


in which 
A=TAT!, B=TB, C=CT™!. 


In the case of a nonlinear system, a static state feedback control law is a control law of the form 
u=a(x)+B(x)v, (46.6) 


where v represents a new control input and B(x) is assumed to be nonzero for all x. Moreover, nonlinear 
changes of coordinates are considered, i.e., transformations of the form 


z= P(x) (46.7) 
where z is the new state vector and ®(x) represents a (n-vector-valued) function of n variables, 


1 (x) 1 (x1, 25° ++ s Xn) 


(x) = #24) | bole xa.-+ + 0) 


n(x) n(x, X25° °° Xn) 


with the following properties: 


1. (x) is invertible; i.e., there exists a function ®~!(z) such that 
O'(O(x))=x,  (@71(z)) =z 


for all x € R” and all z € R”. 
2. (x) and 6~!(z) are both smooth mappings i-e., continuous partial derivatives of any order exist 
for both mappings. 
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A transformation of this type is called a global diffeomorphism. The first property is needed to guarantee 
the invertibility of the transformation to yield the original state vector as 


x= @71(z) 


while the second one guarantees that the description of the system in the new coordinates is still a 
smooth one. 

Sometimes a transformation possessing both these properties and defined for all x is hard to find and 
the properties in question are difficult to check. Thus, in most cases, transformations defined only in a 
neighborhood ofa given point are of interest. A transformation of this type is called a local diffeomorphism. 
To check whether a given transformation is a local diffeomorphism, the following result is very useful. 


Proposition 46.1: 


Suppose P(x) is a smooth function defined on some subset U of R". Suppose the Jacobian matrix 


0o1 991 01 

Ox) Ox2 =s} OXn 

0b2 02 Ob2 
a® = Oxy Ox _ Oxn 
Ox 

Aon 9dn On 

Ox) Ox2 _ Oxy 


is nonsingular at a point x = x°. Then, for some suitable open subset U° of U, containing x°, (x) defines 
a local diffeomorphism between U® and its image ®(U°). 


The effect of a change of coordinates on the description of a nonlinear system can be analyzed as 
follows. Set 
z(t) = ®(x(t)) 


and differentiate both sides with respect to time to yield 


SC eo a le 
a aay a ae 


O® 
Ox (f (x(t) + g(x(t))). 
Then, expressing x(t) as ®~—!(z(t)), one obtains 

2(t) =f (z(t)) + H(z(t))u(t) 


y(t) = h(z(t)) 


where 


~ o® 
eS (FI) oa 


fy O® 
s@) . (S20) x=®@-l(z) 


H(z) = (H) 14: 


The latter are the formulas relating the new description of the system to the original one. 
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Given the nonlinear system of Equation 46.1, the problem of feedback linearization consists of finding, 
if possible, a change of coordinates of the form of Equation 46.7 and a static state feedback of the form of 
Equation 46.6 such that the composed dynamics of Equations 46.1 through 46.6, namely the system 


x= f(x) + g(x)a(x) + g(x)B(x)v, (46.8) 


expressed in the new coordinates z, is the linear and controllable system 


Zy = 22 
2 = 23 
Zn-1 = Zn 
Zi V3 


46.2 Normal Forms of Single-Input Single-Output Systems 


Single-input single-output nonlinear systems can be locally given, by means of a suitable change of 
coordinates in the state space, a “normal form” of special interest, in which several important properties 
can be put in evidence and which is useful in solving the problem of feedback linearization. In this section, 
methods for obtaining this normal form are presented. 

Given a real-valued function of x = (x1,...,Xn) 


A(x) = A(T... Xn) 
and a (n-vector)-valued function of x 


fi, winos Xn) 
fx)= = 
Fn(%1,-- +> Xn) 


we define a new real-valued function of x, denoted Lyi(x), in the following way 


"On 
Liga l ia Gab=>., 5, fie. Xn). 


i=1 


On ( On On 
Ox \ ax; Oxn 


the function Ly(x) can be expressed in the simple form 


Setting 


On 
Lyd) = Bal 


The new function L(x) thus defined is sometimes called the derivative of \(x) along f(x). Repeated 
use of this operation is possible. Thus, for instance, by differentiating \(x) first along f(x) and then along 
g(x), we may construct the function 

OLpr 
LgLy d(x) = Fy 8 
or, differentiating k times (x) along f(x), we may construct a function recursively defined as 
jog 


LEX) = , —f (2). 


With the help of this operation, we introduce the notion of relative degree of a system. 
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Definition 46.1: 


The single-input single-output nonlinear system 


x=f(x)+g(x)u 
y = h(x) 
has relative degree r at x° if: 
1. LgLih(x) = 0 for all x in a neighborhood of x® and allk <r —1. 
2. Lge 'h(x°) #0. 


Example 46.1: 


Consider the system 


0 exp(x2) 
x= ae +sinx2 | + 1 u (46.9) 
—x2 0 


y=h{x)=x3. (46.10) 
For this system we have 
Ooh 
—=( 0 1), lgh)=0, Leh) =— 
ax ( ), Lgh(x) ph(x) = —x2 
O(Lfh) 
=(0 —1 0), Lglph(x) = —1. 
ae! ), Lghphtx) 
Thus, the system has relative degree 2 at any point x°. However, if the output function were, for 


instance 
y=h(x) =x2 


then Lgh(x) = 1 and the system would have relative degree 1 at any point x°. 


The notion of relative degree has the following interesting interpretation. Suppose the system at some 
time f° is in the state x(t) = x°. Calculate the value of y(t), the output of the system, and of its derivatives 
with respect to time, y(t), fork = 1,2,..., at t = f°, to obtain 


y(t°) = h(x(t°)) = h(e°) 


Ohdx Oh 
aan Dx of Ce) + g(x(t))u(t)) = Leh(x(t)) + Leh(x(t))u(t). 


If the relative degree r is larger than 1, for all ¢ such that x(t) is near x 
have Lgh(x(t)) = 0 and therefore 


y(t) _ 


0 ie., for all t near t = t°, we 


y(t) = Lph(x(t)). 
This yields 


OLfh dx  OL¢h 
(2) f of 
yo ay= Bean ae (f (x(t) + g(x(t))u(t)) 


= LPh(x(t)) + LeLh(ex(t))u(t) 


Again, if the relative degree is larger than 2, for all t near t = t°, we have LgLh(x(t)) = 0 and 


y(t) = L7h(x(2)). 
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Continuing in this way, we get 
y(t) = Lfh(x(t)) 


for all k < randall t near t = f°, and 
yl (t9) = Lrh(x°) as LL *h(x°)u(t®). 


Thus, r is exactly the number of times y(t) has to be differentiated at t = t° for u(t®°) to appear explicitly. 

The above calculations suggest that the functions h(x), Lrh(x),...; Ly *h(x) have a special importance. 
As a matter of fact, it is possible to show that they can be used in order to define, at least partially, a 
local coordinate transformation near x°, where x° is a point such that LL *h(x°) # 0. This is formally 
expressed in the following statement. 


Proposition 46.2: 


Let the system of Equations 46.1 and 46.2 be given and let r be its relative degree at x = x°. Thenr <n. Set 


i (x) = h(x) 
2(x) = Lph(x) 


br (2s) = LEG). 


If r is strictly less than n, it is always possible to find n — r additional functions b,+1(x),..., n(x) such 
that the mapping 
1 (x) 


n(x) 


has a Jacobian matrix that is nonsingular at x° and therefore qualifies as a local coordinates transformation 
in a neighborhood of x°. The value at x° of these additional functions can be fixed arbitrarily. Moreover, it 
is always possible to choose ;+41(X),.-., n(x) in such a way that 


Ledj(x) = 0 


forallr+1<i<nandallx around x?, 


The description of the system in the new coordinates zj = $;(x), 1 <i <n can be derived very easily. 
From the previous calculations, we obtain for z1,..., Z; 


dz, Ob, dx _ Ohdx 
dt Ox dt Ox dt 


= Lph(x(t)) = $2(x(t)) = a2(4) 


dz, Od;—1 dx Ly *h dx 
dt Ox dt 0x dt 


= Ly TA(x(t)) = br(x(#)) = 2-(0). 


For z; we obtain 


dz, 3 
= = LfA(x(t)) + LgLf*h(x(t))u(t). 
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On the right-hand side of this equation we must now replace x(t) by its expression as a function of z(t), 
ie, x(t) = ®!(z(t)). Thus, set 

a(z) = LgLF'h(®*(z)) 

b(z) = Lrh(®~"(z)), 


to obtain 


dz; 
an b(z(t)) + a(z(t))u(t). 


As far as the remaining coordinates are concerned, we cannot expect any special structure for the 


corresponding equations. If ,+1(x),..-, n(x) have been chosen so that Lyo;(x) = 0, then 


dzj 
dt 


= ae (f (x(t) + g(x(t))u(t)) = Lp di(x(t)) + Le di(x(t))u(t) = Ly oi(t). 
Setting 

gi(z) = Lyoi(® (z)) 
for r+1<i<n, the latter can be rewritten as 


dz; 
ap qi(z(t)). 


Thus, in summary, the state space description of the system in the new coordinates is as follows 


ZI =22 
2 = 23 
Zp-1 = Zr 


2, = b(z)+ a(z)u 
Zr = 4r+1(Z) 
Zn = qu(z). 


In addition to these equations, one has to specify how the output of the system is related to the new state 
variables. Since y = h(x), it is immediately seen that 


y= 


The equations thus defined are said to be in normal form. Note that at point z° = &(x°), a(z°) £0 by 
definition. Thus, the coefficient a(z) is nonzero for all z in a neighborhood of z°. 


Example 46.2: 


Consider the system of Equations 46.9 and 46.10. In order to find the normal form, we set 
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We now have to find a function 3(x) that completes the coordinate transformation and is such 
that Lgp3(x) = 0, i.e, 


The function 


3(x) = 1+x1 — exp(x2) 


satisfies the equation above. The transformation z = ®(x) defined by the functions $4 (x), 2(x), and 
3(x) has a Jacobian matrix 


which is nonsingular for all x, and ®(0) = 0. Hence, z = ®(x) defines a global change of coordinates. 
The inverse transformation is given by 


X1 =Z3 — 1+ exp(—z?) 
x2 = —Z2 


xX3=2Z1. 
In the new coordinates the system is described by 


zy =2Z2 
Z2 =(1-—23— exp(—z))? +sinz2 —u 


Z3 = exp(—Z2)(sin zz — (z3 —14+ exp(—z))?) . 


These equations describe the system in normal form and are globally valid because the coordinate 
transformation we considered is global. 


46.3 Conditions for Exact Linearization via Feedback 


In this section, conditions and constructive procedures are given for a single-input single-output nonlinear 
system to be transformed into a linear and controllable system via change of coordinates in the state space 
and static state feedback. 

The discussion is based on the normal form developed in the previous section. Consider a nonlinear 
system having at some point x = x° relative degree equal to the dimension of the state space, ie., 7 =n. In 
this case, the change of coordinates required to construct the normal form is given by the function h(x) 
and its first n — 1 derivatives along f(x), i.e., 


1 (x) h(x) 


dn(x)) — \LF *AC) 


(46.11) 


No additional functions are needed to complete the transformation. In the new coordinates 


a= oi(x)=Le'(x) 1lsisn 
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the system is described by equations of the form: 


Z| = 22 
2 = 23 
Zn-1 = Zn 


Zn = b(z) + a(z)u 


where Z = (21, Z2,..-; Zn). Recall that at the point z° = (x), and hence at all z ina neighborhood of 2°, 
the function a(z) is nonzero. 
Choose now the following state feedback control law 


u= ah pe +y), (46.12) 
a(z) 


which indeed exists and is well defined in a neighborhood of z°. The resulting closed-loop system is 
governed by the equations 


n=h (46.13) 
tn = 23 (46.14) 
(46.15) 

Zn—1 = Zn (46.16) 
in = Vs (46.17) 


ie., it is linear and controllable. Thus we conclude that any nonlinear system with relative degree n at 
some point x° can be transformed into a system that is linear and controllable by means of (1) a local 
change of coordinates, and (2) a local static state feedback. 

The two transformations used in order to obtain the linear form can be interchanged: one can first 
apply a feedback and then change the coordinates in the state space, without altering the result. The 
feedback needed to achieve this purpose is exactly the feedback of Equation 46.12, but now expressed in 
the x coordinates as 


u= oa ™ +yv). (46.18) 
Comparing this with the expressions for a(z) and b(z) given in the previous section, one immediately 
realizes that this feedback, expressed in terms of the functions f(x), g(x), h(x), which characterize the 
original system, has the form 
u= ————(-LF(x)+v 
1g). if ( )+¥) 
An easy calculation shows that the feedback of Equation 46.18, together with the same change of coordi- 
nates used so far (Equation 46.11), exactly yields the same linear and controllable system. 
If x° is an equilibrium point for the original nonlinear system, i.e., if f(x°) = 0, and if also h(x°) = 0, 
then 
$1(x°) = A(x°) =0 
and 
i-l 


(yO) SF 0) _ 
se aera f(x") =0 


for all 2<i<n, so that z° = ®(x°) = 0. Note that a condition like h(x°) = 0 can always be satisfied 
by means of a suitable translation of the origin of the output space. Thus, we conclude that, if x° is 
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an equilibrium point for the original system, and this system has relative degree n at x°, there is a 
feedback control law (defined in a neighborhood of x°) and a coordinate transformation (also defined in 
a neighborhood of x’) that change the system into a linear and controllable one, defined in a neighborhood 
of the origin. 

New feedback controls can be imposed on the linear system thus obtained; for example, 


v=Kz 


where 
K=(k kn +e kn) 


can be chosen to meet some given control specifications, e.g., to assign a specific set of eigenvalues or 
to satisfy an optimality criterion. Recalling the expression of the zjs as functions of x, the feedback in 
question can be rewritten as 


v= kyh(x) + koLph(x) +--+ + ky Ly‘ h(x) 


i.e., in the form of a nonlinear feedback from the state x of the original description of the system. 

Up to this point of the presentation, the existence of an “output” function h(x) relative to which the 
system of Equations 46.1 and 46.2 has relative degree exactly equal to n (at x°) has been key in making 
it possible to transform the system into a linear and controllable one. Now, if such a function h(x) is 
not available beforehand, either because the actual output of the system does not satisfy the conditions 
required to have relative degree n or simply because no specific output is defined for the given system, 
the question arises whether it is possible to find an appropriate h(x) that allows output linearization. This 
question is answered in the remaining part of this section. 

Clearly, the problem consists of finding a function, h(x), satisfying the conditions 


Lgh(x) = LgLph(x) = ++» = LgL?*h(x) = 0 


for all x near x° and 
LL? *h(x°) # 0. 

We shall see that these conditions can be transformed into a partial differential equation for h(x), 
for which conditions for existence of solutions as well as constructive integration procedures are well 
known. In order to express this, we need to introduce another type of differential operation. Given two 
(n-vector)-valued functions of x = (x1,...,Xn), f(x) and g(x), we define a new (n-vector)-valued function 
of x, denoted [f, g](x), in the following way 


OB es OF 
Lf, g(x) = axl 9x8) 


Og of 


where — and — are the Jacobian matrices of g(x) and f(x), respectively. The new function thus defined 


x x 
is called the Lie product or Lie bracket of f (x) and g(x). The Lie product can be used repeatedly. Whenever 
a function g(x) is “Lie-multiplied” several times by a function f(x), the following notation is frequently 
used 


adrg(x) =[f, gl) 
ee =[f, Uf gll() 


ates) [f, adi ‘g](x) (x). 


We shall see now that the conditions a function h(x) must obey in order to be eligible as “output” of a 
system with relative degree n can be re-expressed in a form involving the gradient of h(x) and a certain 


Feedback Linearization of Nonlinear Systems 46-11 


number of the repeated Lie products of f(x) and g(x). For, note that, since 
Lip gi h(x) = Le Lg h(x) — Lg Lf h(x), 


if Lgh(x) = 0, the two conditions Lj ¢,¢)h(x) = 0 and LgLyh(x) = 0 are equivalent. Using this property 
repeatedly, one can conclude that a system has relative degree n at x° if and only if 


Ligh (x) = Ladzgh(2) = + = Lygr-2gh(x) = 0 


for all x near x°, and 
0 
Lage ighle ) #0. 


Keeping in mind the definition of derivative of h(x) along a given (n-vector)-valued function, the first set 
of conditions can be rewritten in the following form 


7 (s@) adjg(x) ++: ad?~*g(x)) =0. (46.19) 


This partial differential equation for h(x) has important properties. Indeed, if a function h(x) exists such 
that 
Lgh(x) = Laagh(x) = +++ = Lgr-agh(x) =0 


for all x near x°, and 
0 
Lagrighle ) #0, 


then necessarily the n vectors 
g(x°) adyg(x°) wee ad; *g(x°) ad 'g(x°) 
must be linearly independent. So, in particular, the matrix 
(g(*) adgg(x) +: adj~*g(x)) 


has rank n — 1. The conditions for the existence of solutions to a partial differential equation of the form 
of Equation 46.19 where the matrix 


(g() adgg(x) +>- adj~*g(x)) 


has full rank are given by the well-known Frobenius’ theorem. 


Theorem 46.1: 


Consider a partial differential equation of the form 


h 
7" (Ki) XW) XO) =0, 
in which X\(x),--- ,X(x) are (n-vector)-valued functions of x. Suppose the matrix 
(Xi(x) X2(x) +++ Xe(x)) 


has rank k at the point x = x°. There exist n — k real-valued functions of x, say h(x), ...,hn—4(x), defined 
in a neighborhood of x°, that are solutions of the given partial differential equation, and are such that the 
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Jacobian matrix 
Oh, 
Ox 


Ohyn_k 
Ox 


has rank n—k at x = x° if and only if, for each pair of integers (i,j), 1 < i,j < k, the matrix 
(Xi(x) Xa(x) +++ Xe(x) (Xj, Xj](x)) 
has rank k for all x in a neighborhood of x°. 


Remark 46.1 


A set of k (n-vector)-valued functions {Xj(x), ...,X,(x)}, such that the matrix: 


(Xi(x) X2(x) +++ Xe(x)) 


0 


has rank k at the point x = x°, is said to be involutive near x? if, for each pair of integers (i,j), 1 < i,j <k, 


the matrix 
(Xi(x) Xa(x) +++ Xx(x) LX, Xj](x)) 


still has rank k for all x in a neighborhood of x°. Using this terminology, the necessary and sufficient 
condition indicated in the previous theorem can be simply referred to as the involutivity of the set 
{Xi(x), eas » X4(x)}. 

The arguments developed thus far can be summarized formally as follows. 


Proposition 46.3: 


Consider a system: 
x=f(x)+g(x)u 


There exists an “output” function h(x) for which the system has relative degree n at a point x° if and only if 
the following conditions are satisfied: 


1. The matrix 


(g(x") adpg(x°) ++ ad? *g(x°) adj~'g(x")) 


has rank n. 


2. The set { g(x), adyg(x), cag ad" g(x)} is involutive near x°. 


In view of the results illustrated at the beginning of the section, it is now possible to conclude that 
conditions 1 and 2 listed in this statement are necessary and sufficient conditions for the existence of a 
state feedback and of a change of coordinates transforming, at least locally around the point x°, a given 
nonlinear system of the form 


x=f(x)+ g(x)u 


into a linear and controllable one. 
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Remark 46.2 


For a nonlinear system whose state space has dimension n = 2, condition 2 is always satisfied since 
[g,g](x) = 0. Hence, by the above result, any nonlinear system whose state space has dimension n = 2 
can be transformed into a linear system, via state feedback and change of coordinates, around a point x° 
if and only if the matrix 

(g(x°) adgg(x°)) 
has rank 2. If this is the case, the vector g(x°) is nonzero and it is always possible to find a function 
h(x) = h(x1, x2), defined locally around x°, such that 


x)= X1,X%2) + X1,X%2.) = 0. 
Ox Oxy eee Ox aii 


If a nonlinear system of the form of Equations 46.1 and 46.2 having relative degree strictly less than 
n meets requirements 1 and 2 of the previous proposition, there exists a different “output” function, say 
k(x), with respect to which the system has relative degree exactly n. Starting from this new function, it is 
possible to construct a feedback u = a(x) + B(x)v and a change of coordinates z = ®(x), that transform 
the system 

x= f(x) +g(x)u 
into a linear and controllable one. However, in general, the real output of the system expressed in the new 
coordinates 
y=h(&*(z)) 


is still a nonlinear function of the state z. Then the question arises whether there exist a feedback and a 
change of coordinates transforming the entire description of the system, output function included, into 
a linear and controllable one. The appropriate conditions should include the previous ones with some 
additional constraints arising from the need to linearize the output map. For the sake of completeness, a 
possible way of stating these conditions is given hereafter. 


Proposition 46.4: 


Let the system of Equations 46.1 and 46.2 be given and let r be its relative degree at x = x°. There exist a 
static state feedback and a change of coordinates, defined locally around x°, so that the system is transformed 
into a linear and controllable one 

x= Ax+ Bu 

y=Cx 
if and only if the following conditions are satisfied: 


1. The matrix 
(g(") adgg(x°) ++ ad? *g(x°) adj~‘g(x")) 
has rank n. 
2. The (n-vector)-valued functions defined as 


r 


F(x) =f (x) ne) (x) Hx) = 
Fay= fee Gene 


1 
Lgl} th(x)® @) 


are such that 
[aaiz.aaz] =0 
for all pairs (i,j) such that 0 <i,j <n. 
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Example 46.3: 


Consider the system of Equations 46.9 and 46.10. In order to see if this system can be transformed 
into a linear and controllable system via static state feedback and change of coordinates, we have to 
check conditions 1 and 2 of Proposition 46.3. We first compute ad¢g(x) and ad? g(x): 


exp(x)(x7 + sin x2) 


ad g(x) = —2x, exp(x2) — cos x2 


1 
exp(x) (x? + sin x2)(x? + sin xz + cos x2) 
ad? q(x) = x?(sin X2 — 4x1 exp(x2)) + 1 — 4x7 exp(x2) sin x2 + 2x1 exp(x2) cos x2 
—2x1 exp(x2) — cos x2. 


The matrix 
(0) ad;g(x) ad?-g(x)) 


has rank 3 at all points x where its determinant, an analytic function of x, is different from zero. Hence, 
condition 1 is satisfied almost everywhere. Note that at point x = 0 the matrix 


(0%) ad;g(x) ad?g(x)) 


has rank 2, and this shows that condition 1 is not satisfied at the origin. 
The product [g, ad¢g](x) has the form 


4x1 exp(2x2) + exp(X2)(x? + sin xz + 2c0s x2) 


[g, ad¢g}(x) = sin xp — 2 exp(2x>) — 2x, exp(x3) 
0 


Then one can see that the matrix 
(gx) adgg(x) [g, ad¢g](x)) 
has rank 2 at all points x for which its determinant 
exp(x2)(2 exp(2x2) + 6x1 exp(x2) + 2 cos x2 + x?) 


is zero. This set of points has measure zero. Hence, condition 2 is not satisfied at any point x of the 
state space. 


In summary, the system of Equations 46.9 and 46.10 satisfies condition 1 almost everywhere but does 
not satisfy condition 2. Hence, it is not locally feedback linearizable. 


Example 46.4: 


Consider the system 


X3 —X2 0 
x= 0 + | exp(x;) J u 
x3+x2]— \expox) 
Y=X2. 


This system has relative degree 1 at all x since Lgh(x) = exp(x;). It is easily checked that conditions 
1 and 2 of Proposition 46.3 are satisfied. Hence, there exists a function h(x) for which the system has 
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relative degree 3. This function has to satisfy 
Oh 
Ox 


A solution to this equation is given by 


(g(x) adgg(x)) = 0. 


h(x) = x). 


The system can be transformed into a linear and controllable one by means of the static state 
feedback 


—L3 A(x) +V  2x1X2 — 2x1x3 — X3 —x? +v 
7 Lgl?h(x) ~ exp(x7) 


and the coordinates transformation 
Z1 =h(x) =x) 
Z2 = L¢h(x) = x3 — x2 
Z3= L? h(x) =X3 +x? - 
The original output of the system y = x9 is a nonlinear function of z: 
- 2 
yY =—-2Z2+23-2Z7. 


To determine whether the entire system, output function included, can be transformed into a linear 


and controllable one, condition 2 of Proposition 46.4 should be checked. Since L¢h(x) = 0, f(x) = f(x) 
0 

and g(x) = | 1 |. Easy calculations yield 
1 


—2x;-1 


One can check that [ad2g, ad2g] #0. Hence, condition 2 of Proposition 46.4 is not satisfied. 
Therefore, the system with its output cannot be transformed into a linear and controllable one. 
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47.1 Limit Sets 


In linear system theory, if the variables which characterize the behavior of a system are either constant 
or periodic functions of time, the system is said to be in steady state. In a stable linear system, the steady 
state can be seen as a limit behavior, approached either as the actual time t tends to +00 or, alternatively, 
as the initial time to tends to —oo (the two viewpoints being in fact equivalent). For a general nonlinear 
dynamical system, concepts yielding to a notion of steady-state repose on certain fundamental ideas 
dating back to the works of H. Poincaré and G.D. Birkhoff.* In particular, a fundamental role is played by 
the concept of w-limit set of a given point, which is defined as follows. Consider an autonomous ordinary 
differential equation 


x= f(x), (47.1) 


in which x € R”. Suppose that f(x) is locally Lipschitz. Then, it is well-known that, for any xo € R", the 
solution of Equation 47.1 with initial condition x(0) = xo, denoted in what follows by x(t, xo), exists on 
some open interval of the point t = 0 and is unique. 
Suppose that, for some x9, the solution x(t, xo), is defined for all t > 0, that is, for all forward times. 
A point x is said to be an w-limit point of the trajectory x(t, xo) if there exists a sequence of times {f;}, 
with limz_, 9 t, = 00, such that 
lim x(ty, xo) =x. 
k->0o 
The w-limit set of a point x9, denoted w(xo), is the union of all w-limit points of the trajectory x(t, xo). 
It is obvious from this definition that an w-limit point is not necessarily a limit of x(t, x9) as t > 00, 
since the solution in question may not admit any limit as t > oo. However, it happens that if the motion 
x(t, xo) is bounded, then x(t, xo) asymptotically approaches the set w(x). In fact, the following property 
holds [1, p. 198]. 


* Relevant, in this regard, are the concepts introduced by G.D. Birkhoff in his classical 1927 essay, where he asserts that 


“with an arbitrary dynamical system ... there is associated always a closed set of ‘central motions’ which do possess this 
property of regional recurrence, towards which all other motions of the system in general tend asymptotically” [1, p. 190]. 


ari 
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Lemma 47.1: 


Suppose x(t, xo) is bounded in forward time, that is, there isa M > 0 such that ||x(t, xo)|| < M for allt > 0. 
Then, w(xo) is a nonempty compact connected set, invariant under the dynamics equation 47.1.* Moreover, 
the distance of x(t, xo) from o(xo) tends to 0 as t > oo.? 


One of the features of the set w(xo), highlighted in this Lemma, is that this set is invariant for 
Equation 47.1. Hence, the set w(xo) is filled by trajectories of Equation 47.1 which are defined for all 
backward and forward times, and also bounded, because so is the set w(xo). The other relevant feature is 
that x(t, xo) asymptotically approaches the set w(x) as t — 00, in the sense that the distance of the point 
x(t, xo) from the set w(x) tends to 0 as t > oo. 

Suppose now that the initial conditions of Equation 47.1 range over some set B and that x(t, xo) 
is bounded, in forward time, for any xo € B. Since any trajectory which is bounded in forward time 
asymptotically approaches its own w-limit set w(xo), it is concluded that any trajectory obtained by 
picking xo in B asymptotically approaches the set 


Y@)= L) oo). 


xoEB 


The set in question is filled by trajectories of Equation 47.1 which are bounded in forward and backward 
time. Since the set y(B) is approached asymptotically by any trajectory with initial condition in B, it 
seems plausible to regard the set of all trajectories evolving in p(B) as the set of steady-state “behaviors” 
of Equation 47.1. There is, however, a problem in taking this as the definition of steady-state behavior of 
a nonlinear system: the convergence of x(t, xo) to y(B) is not guaranteed to be uniform in xo, even if the 
set B is compact (see, e.g., [2]). 

Uniform convergence to the steady state, which is automatically guaranteed in the case of linear 
systems, is an important feature to be kept in extending the notion of steady state from linear to nonlinear 
systems. In fact, the notion of steady-state would lose much of its practical relevance if the convergence 
were not uniform, that is, if the time needed to get within an e-distance from the steady state could grow 
unbounded when the initial state is varied (even when the latter is picked within a fixed bounded set). 
Thus, the set y(B) is not a good candidate for a definition of steady state in a nonlinear system. There 
is a larger set, however, which does have this property of uniform convergence. This set, known as the 
w limit set of a set, is defined as follows. Suppose that, for all xo € B, the trajectory x(t, xo) is defined for 
all t > 0. The w-limit set of B, denoted «(B), is the set of all points x for which there exists a sequence of 
pairs {x,, t,}, with x, € Band limp_,oo tk = 00 such that 


lim x(ty, xp) =x. 
k->0oo 


It is clear from this definition that if B consists of only one single point xo, all x;’s in the definition above 
are necessarily equal to xo and the definition in question reduces to the definition of w-limit set of a 
point. It is also clear from this definition that, if for some xo € B the set w(xo) is nonempty, all points of 
w(xo) are points of w(B). In fact, all such points have the property indicated in the definition, if all the 
x,’s are taken equal to xp. Thus, in particular, if all motions with x) € B are bounded in forward time, 
W(B) C w(B). However, the converse inclusion is not true in general. The simplest example in which this 


* A set S is invariant for Equation 47.1 if, for any xg € S, the solution x(t, xq) exists for all t € R and x(t, xq) € Sforallt eR. 


+ The distance of a point x € R” from a set S C R", written as dist(x, S), is defined as infyes lly — xll- 
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fact can be checked is the the case of a stable Van der Pol oscillator 


x=y, 
yo —x—e(1—x*)y €>0. 


If B is a disc of sufficiently large radius, centered at (x, y) = (0,0), the set W(B) consists of the union of 
the (unstable) equilibrium at (x, y) = (0,0) and of the (stable) limit cycle. On the contrary, the set w(B) 
consists of all points on the limit cycle and of all points inside this limit cycle. 

The counterpart of Lemma 47.1 is the following result, which characterizes the relevant features of the 
concept of the w-limit set of a set [3, p. 8]. 


Lemma 47.2: 


Let B be a nonempty bounded subset of R" and suppose there is a number M such that ||x(t, xo) || < M for 
allt > 0 and all xp € B. Then w(B) is a nonempty compact set, invariant under Equation 47.1. Moreover, 
the distance of x(t, xo) from w(B) tends to 0 as t > 0, uniformly in xo € B. If B is connected, so is w(B). 


As in the case of the set (B), it is seen that the set w(B) is filled with trajectories which are defined 
for all backward and forward times, and bounded. But, above all, it is seen that the set in question is 
uniformly approached by trajectories with initial state x9 € B, a property that the set (B) does not have. 
The set w(B) asymptotically attracts, as t > 00, all motions that start in B. Since the convergence to 
«(B) is uniform in x9, it is also true that, whenever w(B) is contained in the interior of B, the set w(B) is 
asymptotically stable, in the sense of Lyapunov, that is, for every number ¢ > 0 there is a number 8 > 0 
such that, if the distance of xo from w(B) is less than 8, then the distance of x(t, xo) from w(B) is less than € 
for all t > 0. This property is very important in nonlinear feedback design, because it is a key property in 
establishing the existence of Lyapunov functions, as often required in most results concerning the design 
of stabilizing feedback laws. 


47.2 The Steady-State Behavior of a System 


Consider an autonomous finite-dimensional system 
x =f (x). (47.2) 


with initial conditions in a closed subset X C R”, and suppose the set X is forward invariant, that is, for 
any initial condition xp € X, the solution x(t, x9) exists for all t > 0 and x(t, xo) € X for allt > 0. 

The motions of this system are said to be ultimately bounded if there is a bounded subset B with the 
property that, for every compact subset X of X, there is a time T' > 0 such that x(t,xo) € B for all t > T 
and all xo € X. In other words, the motions are ultimately bounded if any admissible trajectory in finite 
time (this finite time possibly being dependent on the chosen initial condition) enters a bounded set B, 
and remains in this set for all future times. 

If the motions of a system are ultimately bounded, in particular all trajectories with initial conditions 
in B are bounded. As a consequence, the limit set w(B) is nonempty. Since all trajectories with initial 
conditions in X eventually enter the set B, the set w(B)—which by Lemma 47.2 attracts all trajectories 
with initial condition in B—also attracts all trajectories with initial conditions in X. It is therefore natural 
to regard w(B) as a set to which all admissible trajectories of Equation 47.2 converge. While the set B, 
in the definition of ultimate boundedness, is not uniquely characterized, it is easy to see that the set 
«(B) is, on the contrary, a uniquely defined object. In fact, it is possible to prove that if the motions of 
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Equation 47.2 are ultimately bounded and if B’ # B is any other bounded subset with the property that, 
for every compact subset X of X, there is a time T > 0 such that x(t, xo) € B’ for all t > T and all xp € X, 
then w(B’) = (B) (see, e.g., [2]). 

In view of these properties, it is concluded that if the motions of a system are ultimately bounded, any 
trajectory asymptotically approaches a uniquely defined compact invariant set, the set w(B). The latter is 
nonempty, compact, and invariant. In other words, any trajectory of the system approaches a nonempty 
set, which is in turn filled by other trajectories, which are defined and bounded in forward and backward 
time. Thus it is natural to look at any of such trajectories as a steady-state trajectory and to regard the set 
«(B) as the set in which the steady-state behavior of the system takes place. This leads to the following 
definition (see [2]). 


Definition 47.1: 


Suppose the motions of system (Equation 47.2), with initial conditions in a closed and forward invariant 
set X, are ultimately bounded. A steady-state motion is any motion with initial condition x(0) € w(B). 
The set w(B) is the steady-state locus of Equation 47.2 and the restriction of Equation 47.2 to w(B) is the 
steady-state behavior of Equation 47.2. 


47.3 The Steady-State Response 


The definition given in the previous section recaptures the classical notion of steady state for linear 
systems and provides a powerful tool to deal with similar issues in the case of nonlinear systems. We 
discuss in what follows a number of relevant cases. 

Consider an n-dimensional, single-input, asymptotically stable linear system 


zZ=Az+Bu, (47.3) 


forced by the harmonic input u(t) = uo sin(Qt + oo). It is well-known that, regardless of what the initial 
condition is, the response z(t) converges to a unique, well-defined, steady-state response, a periodic 
function of period 2m/Q. A simple (geometric) method to determine such response consists in viewing 
the forcing input u(t) as provided by an autonomous “signal generator” of the form 


w=Sw u=Qw 


0 Q 
ce D Q=(1 0) 


and in analyzing the state behavior of the associated “augmented” system 


in which 


w = Sw, 
. (47.4) 
Z= BQw+ Az. 


Since the matrices S and A do not have common eigenvalues, the Sylvester equation IIS = ATI + BQ 
has a unique solution. The augmented system possesses two complementary invariant subspaces: a stable 
invariant subspace and a center invariant subspace. The former is the set of all pairs (w, z) in which w = 0, 
while the latter is the graph of a linear map 
m: R? > R" 
wr> IIw. 


All trajectories of Equation 47.4 approach the center invariant subspace as f — oo, and hence the limit 
behavior of Equation 47.4 is determined by the restriction of its motions to this invariant subspace. As a 
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consequence, the steady-state response of Equation 47.3 to the periodic input u(t) = up sin(Qt + oo) is 
given by 


x(¢) =Mw(t) =11 (‘° ae) 


ug cos(Qt + oo) 


Revisiting this analysis from the viewpoint of the more general notion of steady state introduced above, 
let W C R? bea set of the form 


W ={weR?: ||wl| <c} (47.5) 


in which c is a fixed number, and suppose the set of initial conditions for Equation 47.4 is W x R”. This 
is the case when the problem of evaluating the periodic response of Equation 47.3 to harmonic inputs 
whose amplitude does not exceed a fixed number c is addressed. Note that the set W is compact and 
invariant for the upper subsystem of Equation 47.4. 

The set W x R” is closed and forward invariant for the full system Equation 47.4 and, moreover, since 
the lower subsystem of Equation 47.4 is a linear asymptotically stable system driven by a bounded input, it 
is immediate to check that the motions of system Equation 47.4, with initial conditions taken in W x R", 
are ultimately bounded. In particular, 


o(B) = {(w,z) € R? x R": we W,z= Tw}, 


that is, w(B) is the graph of the restriction of the map 1 to the set W. The restriction of Equation 47.4 
to the invariant set w(B) characterizes the steady-state behavior of Equation 47.3 under the family of all 
harmonic inputs of fixed angular frequency w, and amplitude not exceeding c. 

A similar result holds if u(t) is provided by a nonlinear “signal generator” of the form 


w=s(w), u=q(w). (47.6) 
In fact, consider an augmented system of the form 


w = s(w), 
(47.7) 
Z = Bq(w)+ Az, 
in which w € W CR’, with W a compact set, invariant for the the upper subsystem of Equation 47.7. 
Suppose, as before, that the matrix A has eigenvalues with negative real part. 

As in the previous example, since the lower subsystem of Equation 47.7 is a linear asymptotically 
stable system driven by the bounded input u(t) = q(w(t, wo)), the motions of system Equation 47.7, with 
initial conditions taken in W x R", are ultimately bounded. A simple calculation (see [2]) shows that the 
steady-state locus of Equation 47.7 is the graph of the map 


tm: W— R" 
wtr> m(w), 
defined by 
0 
m(w) = jim e A°Bq(w(t, w)) dt. (47.8) 
—>0o 


Asa consequence, it is concluded that the steady-state response of Equation 47.3 to an input u(t) produced 
by a signal generator of the form (Equation 47.6) can be expressed as z(t) = m(w(t)). 
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There are various ways in which the result discussed in the previous example can be generalized. For 
instance, it can be extended to describe the steady-state response of a nonlinear system 


Z=f(z,u) (47.9) 


to an input provided by a nonlinear signal generator of the form (Equation 47.6), if system (Equation 47.9) 
is input-to-state stable.* In this case, in fact, the composition of Equations 47.9 and 47.6 


w = s(w), 


; (47.10) 
z=f(z,q(w)), 


with initial conditions (wo, zo) € W x R" is a system whose motions are ultimately bounded and hence a 
well-defined steady-state locus exists. 

A common feature of the two examples (Equations 47.4 and 47.7) is the fact that the steady-state locus 
of the system can be expressed as the graph of a map, defined on the set W. This means that, so long 
as this is the case, the system has a unique well-defined steady-state response to the input u(t) = q(w(t)), 
expressible as z(t) = m(w(t)). Of course, in general, this may not be the case and the global structure of the 
steady-state locus could be very complicated. In particular, the set w(B) may fail to be the graph of a map 
defined on W and multiple steady-state responses to a given input may occur. This is the counterpart—in 
the context of forced motions—of the fact that, in general, a nonlinear system may possess multiple 
equilibria. In these cases, the steady-state response is determined not only by the forcing input, but also 
by the initial state of the system to which the input is applied. 

Even though, in general, uniqueness of the steady-state response of a system (Equation 47.9) to inputs 
generated by a system of the form (Equation 47.6) cannot be guaranteed, if the set W is compact and 
invariant (as assumed above), for each w € W there is always at least one initial condition z of Equa- 
tion 47.9 such that the pair (w, z) produces a steady-state response. 


Lemma 47.3: 


Let W be a compact set, invariant under the flow of Equation 47.6. Let Z be a closed set and suppose that 
the motions of Equation 47.10 with initial conditions in W x Z are ultimately bounded. Then, the steady 
state locus of Equation 47.10 is the graph of a set-valued map defined on the whole of W. 


This result is particularly useful in establishing certain necessary conditions in the analysis of the 
problem of nonlinear output regulation. 
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48.1 The Problem 


A classical problem in control theory is to impose, via feedback, a prescribed steady-state response to every 
external command in a given family. This may include, for instance, the problem of having the output 
of a controlled plant asymptotically track any prescribed reference signal in a certain class of functions 
of time, as well as the problem of having this output asymptotically reject any undesired disturbance in a 
certain class of disturbances. In both cases, the issue is to force a suitably defined tracking error to zero, as 
time tends to infinity, for every reference output and every undesired disturbance ranging over prescribed 
families of functions of time. 

Generally speaking, the problem can be cast as follows. Consider a finite-dimensional, time-invariant, 
nonlinear system modeled by equations of the form 


x = F(w, x, u), 


aia. (48.1) 


in which x € R” is a vector of state variables, u € R is a vector of inputs used for control purposes, w € R* 
is a vector of inputs that cannot be controlled and include exogenous commands, exogenous disturbances, 
and model uncertainties, and e € R is a vector of regulated outputs that include tracking errors and any 
other variable that needs to be steered to 0. The problem is to design a controller, which receives e(t) 
as input and produces u(t) as output, able to guarantee that, in the resulting closed-loop system, x(t) 
remains bounded and e(t) — 0 as t > ov, regardless of what the exogenous input w(t) actually is. 

The ability to successfully address this problem very much depends on how much the controller is 
allowed to know about the exogenous disturbance w(t). In the ideal situation in which w(t) is available 
to the controller in real time, the design problem indeed looks much simpler. This is, however, only an 
extremely optimistic situation which does not represent, in any circumstance, a realistic scenario. The 
other extreme situation is the one in which nothing is known about w(t). In this, pessimistic, scenario the 
best result one could hope for is the fulfillment of some prescribed ultimate bound for |e(t)|, but certainly 
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not a sharp goal such as the convergence of e(t) to 0. A more comfortable, intermediate, situation is the 
one in which w(t) is only known fo belong to a fixed family of functions of time, for instance, the family 
of all solutions obtained from a fixed ordinary differential equation of the form 


w= s(w) (48.2) 


as the corresponding initial condition w(0) is allowed to vary on a prescribed set. This situation is in fact 
sufficiently distant from the ideal but unrealistic case of perfect knowledge of w(t) and from the realistic 
but conservative case of totally unknown w(t). But, above all, this way of thinking about the exogenous 
inputs covers a number of cases of major practical relevance. There is, in fact, an abundance of design 
problems in which parameter uncertainties, reference command, and/or exogenous disturbances can be 
modeled as functions of time that satisfy an ordinary differential equation. 

The control law is to be provided by a system modeled by equations of the form 


Xe =F e(X3 e), (48 3) 
UP Hee), 


with state x, € R’. The initial conditions x(0) of the controlled plant (Equation 48.1), w(0) of the exosystem 
(Equation 48.2), and x,(0) of the controller (Equation 48.3) are allowed to range over a fixed compact sets 
X CR", W CR’, and X, C R’ respectively. All maps characterizing the model of the controlled plant, of 
the exosystem, and of the controller are assumed to be sufficiently differentiable. 

The problem that is analyzed in this chapter, known as the problem of output regulation (or generalized 
tracking problem or also generalized servomechanism problem), is to design a feedback controller of the 
form (Equation 48.3) so as to obtain a closed-loop system in which all trajectories are bounded and 
the regulated output e(t) asymptotically decays to 0 as t + oo. More precisely, it is required that the 
composition of Equations 48.1, 48.2, and 48.3, that is the autonomous system 


w=s(w), 
x = F(w, x, He(x-, H(w, x)), (48.4) 
Xe = F.(Xe; H(w, x))s 


with output 
e=H(w,x), 


be such that 


¢ The positive orbitof W x X x X, is bounded, that is, there exists a bounded subset S of R* x R” x R” 
such that, for any (wo, x0; Xc,o0) € W x X x X,, the integral curve (w(t), x(t), xc(t)) of Equation 48.4 
passing through (wo, Xo, Xc,) at time f = 0 remains in S for all t > 0. 

¢ lim:-5o0 e(f) = 0, uniformly in the initial condition, that is, for every ¢ > 0 there exists a time 
t, depending only on ¢ and not on (wo, x0, Xc0) € W x X x X¢, such that the integral curve 
(w(t), x(t), xc(t)) of Equation 48.4 passing through (wo, x0, xc) at time t = 0 satisfies |e(t)| <e 
for allt > t. 


Cast in these terms, the problem is readily seen to be equivalent to a problem to design a controller 
yielding a closed-loop system that possesses a steady-state locus* entirely immersed in the set of all (w, x) 
at which the regulated output e = H(w, x) is 0. This being the case, there is no loss of generality in assuming 
from the very beginning that the exosystem is in steady state, which is the case when the compact set W 
is invariant under the dynamics of Equation 48.2. This will be assumed throughout the entire chapter. 


* See Chapter 47 for a definition of steady state in a nonlinear system. 
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Note also that an approach of this kind covers also the case in which the exosystem dynamics admit a 
decomposition of the form 


Ww = $1 (W1, W2), 


w2 = 0, 


in which some of the components of the exogenous input have a trivial dynamics, that is, are constant. 
The elements of w2 comprise any uncertain constant parameters (assumed to range on a compact set) 
affecting the model of the controlled plant (Equation 48.1), as well as the dynamics of the time-varying 
components of w. Thus, solving a design problem cast in these terms provides robustness with respect to 
structured parametric uncertainties in the model of the plant, as well as in the model of the exogenous 
inputs to be tracked and/or rejected. 


48.2 The Case of Linear Systems as a Design Paradigm 


As an introduction, we describe in this section how the problem of output regulation can be analyzed and 
solved for linear systems. This provides in fact an instructive design paradigm that can be successfully 
followed in handling the corresponding general problem. The first step is the analysis of the steady state, 
which in turn entails the derivation of certain necessary conditions. 

Consider the case in which the composition of exosystem (Equation 48.2) and controlled plant (Equa- 
tion 48.1) is modeled by equations of the form 


w = Sw, 
x= Pw+Ax-+ Bu, (48.5) 
e= Qw+Cx, 


and the controller (Equation 48.3) is modeled by equations of the form 


Xe = AcxXe + Bee, 


(48.6) 
u= Cox, + Dee. 
The associated closed-loop system is the autonomous linear system 
w S 0 0 w 
x |= ]P+BD.Q A+BD,.C BC, x]. (48.7) 
Xe B-Q B.C Ac Xc 


If the controller (Equation 48.6) solves the problem of output regulation, the trajectories of 
Equation 48.7 are bounded and, necessarily, all the eigenvalues of the matrix 


A+BD,.C BC, 
B.C Ag 


have negative real part. Since by assumption S has all eigenvalues on the imaginary axis, system 
(Equation 48.7) possesses two complementary invariant subspaces: a stable invariant subspace and a 
center invariant subspace. The latter, in particular, is the graph of a linear map 
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in which I and I], are solutions of the Sylvester equation 


n _ (A+BD.C_ BC,\ (Tl P+BD,.Q 
(= B.C 7) (ne) + B.O : (48.8) 


Any trajectory of Equation 48.7 has a unique decomposition into a component entirely contained in 
the stable invariant subspace and a component entirely contained in the center invariant subspace. The 
former, which asymptotically decays to 0 as t + 00 is the transient component of the trajectory. The latter, 
in which x(t) and x,(t) have, respectively, the form 


x(t)=Tw(t),  x-(t) = Tew(t) (48.9) 


is the steady-state component of the trajectory. 

If the controller (Equation 48.6) solves the problem of output regulation, the steady-state component of 
any trajectory must be contained in the kernel of the map e = Qw + Cx and hence the solution (IT, I.) of 
the Sylvester equation 48.8 necessarily satisfies Q-+ CII = 0. Entering this constraint into Equation 48.8 
it is concluded that if the controller (Equation 48.6) solves the problem of output regulation, necessarily 
there exists a pair (II, I1,) satisfying 


IS = AI1+ BC... + P, 
TI.S = A.M. 
0=CII+Q. 


Setting YW = CI, the first and third equations are more conveniently rewritten in the (controller- 
independent) form 


NS = AIN+BW +P, 


(48.10) 
0=CII+Q, 
in which, of course, V is a matrix satisfying 
W=C.M,, 
ihe (48.11) 
TI.S = A. Te; 


for some choice of IT,, Ac, Cc. The linear equations 48.10 are known as Francis’ equations and the existence 
of a solution pair (TI, YW) is—as shown—a necessary condition for the solution of the problem of output 
regulation [1,10,11]. 

Equations 48.11, from a general viewpoint, could be regarded as a constraint on the component W of 
the solution of Francis’ equations 48.10. However, as an easy calculation shows, this constraint is actually 
irrelevant. In fact, given any pair S, W it is always possible to fulfill conditions like (Equation 48.11). Let 


A) =p 4st oe Hep 


denote the minimal polynomial of S and set 


w 0 1 0 : 0 
WS 0 0 Looe 0 
T= ars , C= E é ; aigts : , T= 0 0 --- 0). 
wsd-2 0 0 0: ys 1 
wsd-l —Sp —Sp —S2 +++ —Sg-y 


Then, it is immediate to check that 
W=IT, 
(48.12) 
TS = OT, 


which is precisely a constraint of the form (Equation 48.11). Note, in particular, that the pair (®, T°) thus 
defined is observable. 
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The relevant role, however, of Equations 48.11 is that they interpret the ability, of the controller, to 
generate the feedforward input necessary to keep the regulated variable identically zero in steady state. In 
steady state—as shown—the state x(t) of the plant and x,(t) of the controller evolve as in Equation 48.9 
and e(t) = 0. Consequently, in steady state the controller is driven by input which is identically zero and 
generates, as output, a control of the form 


Ugs(t) = C.x,(t) = C.T1.w(t) = Wwi(t). 


The latter, as predicated by Francis’ equations, is a control able to force a steady state trajectory of 
the form x(t) = w(t) and consequently to keep e(t) identically zero. The property thus described 
is usually referred to as the internal model property: any controller that solves the problem of out- 
put regulation necessarily embeds a model of the feedforward inputs needed to keep e(t) identically 
zero [2]. 

We proceed now with the design of a control that ensures asymptotic convergence to the required 
steady state. In view of the above analysis, to solve the problem of output regulation it is natural to assume 
that Francis’ equations 48.10 have a solution, and to consider a controller of the form 


u=Tyn+v, 


(48.13) 
y= Oyn4+v, 


where I’ and © satisfy (Equation 48.12) for some T (which, as shown, is always possibile) and where v, v’ 
are additional controls. If these controls vanish in steady state, the graph of the linear map 


x Il 
Wh> (*) I a) w (48.14) 


is, by construction, an invariant subspace of the composite system 


w = Sw, 

x= Pw+Ax+B(n+yv), 

; are ey) (48.15) 
n= On+Vv, 

e=Qw+Cx. 


The regulated variable e vanishes on the graph of Equation 48.14. Thus, if the additional controls v and v’ 
are able to (robustly) steer all trajectories to this invariant subspace, the problem of output regulation is 
solved. 

Let now the states x and 1 of Equation 48.15 be replaced by the differences 


in which case the equations describing the system, by virtue of Equations 48.10 and 48.12, become 


X = AX + BI + By, 
fa one, (48.16) 
e 


To steer all trajectories of Equation 48.15 to the graph of Equation 48.14 is the same as to stabilize the 
equilibrium (x, 7) = (0, 0) of Equation 48.16. 
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A simple design option, at this point, is to set v’ = Gv, and to seek a (possibly dynamic) controller, 
with input e and output v, which (robustly) stabilizes the equilibrium (x, n) = (0, 0) of 


x = AX + BI) + By, 
Pere Gi, (48.17) 
e 


A sufficient condition under which such a robust stabilizer exists is that all zeros of Equation 48.17 have 
negative real part. The zeros of system (Equation 48.17), on the other hand, are the roots of the equation 


A—dI Br B A—dI 0 B 
0 = det 0 ®—)I GI] =det 0 ®—GI-idI G 
G 0 0 C 0 0 
A-?dI B 
= det ( C det(® — GI — iJ), 


and hence it is readily seen that a sufficient condition for the existence of the desired robust stabilizer is 


that all roots of the equation 
A-)dI B 
0 = det ( :, \) (48.18) 


and all eigenvalues of the matrix @ — GI have negative real part. The latter condition can always be 
fulfilled. In fact, as observed earlier, it is always possible to find an observable pair (®, I) which render 
(Equation 48.12) fulfilled for some T. Hence, there always exists a vector G which makes ®— GI’ a 
Hurwitz matrix. In view of this, it is concluded that a sufficient condition for the existence of a robust 
stabilizer for Equation 48.17, once the vector G has been chosen in this way, is simply that all roots of 
Equation 48.18, or—what is the same—all zeros of 


x= Ax-+ Bu, 
(48.19) 
e=Cx 


have negative real part. This condition, on the other hand, also guarantees that the Francis’ equation 48.10 
have a solution. 


48.3 Steady-State Analysis 


Controlling the nonlinear plant (Equation 48.1) by means of the nonlinear controller (Equation 48.3) 
yields a closed-loop system modeled by Equations 48.4. If the problem of output regulation is solved, the 
positive orbit of the set W x X x X; of initial conditions is bounded and hence all trajectories asymptot- 
ically approach a steady-state locus w(W x X x X,). This set is the graph of a (possibly set-valued) map 
defined on W.* To streamline the analysis, we assume that this map is single-valued, that is, that there 
exists a pair of maps x = m(w) and x, = 1(w), defined on W, such that 


w(W x X x X) = {(w, x, x): we Wx = m(w), x = 1(w)}. (48.20) 


This is equivalent to assume that, in the closed-loop system, for each given exogenous input function 
w(t), there exists a unique steady state response, which therefore can be expressed as x(t) = m(w(t)) 
and x,(t) = m,(w(t)). Moreover, for convenience, we also assume that the maps m(w) and 1,.(w) are 


* See Chapter 47 for details. 
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continuously differentiable. This enables us to characterize in simple terms the property that the steady 
state locus is invariant under the flow of the closed-loop system (Equation 48.4). If this is the case, in fact, 
to say that the locus (Equation 48.20) is invariant under the flow of Equation 48.4 is the same as to say 
that m(w) and 1,.(w) satisfy 


3) 

= s(W) = F(w, n(w), He(1-(w), H(w, n(w)))), 

fs Vwe W. (48.21) 
Sw) = F.(1-(w), H(w, x(w))), 


These equations are the nonlinear counterpart of the Sylvester equations 48.8. If the controller solves the 
problem of output regulation, the steady state locus, which is asymptotically approached by the trajectories 
of the closed-loop system, must be a subset of the set of all pairs (w, x) for which H(w, x) = 0 and hence 
the map m(w) necessarily satisfies H(w, 1(w)) = 0. Entering this constraint into Equation 48.21 it follows 
that 


EG) = F(w, n(w), H-(1-(w), 0)), 
Ow 


one s(w) = F.(1-(w), 0), (48.22) 
Ow 


0= H(w, n(w)). 


Proceeding as in the case of linear systems and setting W(w) = H.(1-(w), 0), the first and third Equa- 
tions of 48.22 can be rewritten in controller-independent form as 


On 
Bp ON ee (48.23) 


0=H(w, x(w)), 


These equations, introduced in [3] and known as the nonlinear regulator equations, are the nonlinear 
counterpart of the Francis’ equations 48.10. 
Observe that the map y(w) appearing in Equation 48.23 satisfies 


ww) —_ H,(1-(w), 0), 
48.24 
OE sw) = Fel te(W),0). ee 


These constraints also can be formally rewritten in controller-independent form. In fact, the constraint 
in question simply expresses the existence of an integer d, of an autonomous dynamical system 


H=9(n), neR4 (48.25) 
with output 
u= y(n); (48.26) 


and of a map t: W > R® such that 


ww) = y(t(w)), 
Ot Vwe W. (48.27) 
ay) = 9(t(w)), 

Ww 


These are nonlinear counterparts of the constraints (Equation 48.12). However, while in the case of linear 
systems constraints of this form are irrelevant (i.e., can always be satisfied), in the case of nonlinear systems 
some technical problems arise and the existence of a triplet {¢(-), y(-), t(-)} satisfying Equation 48.27 may 
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require extra hypotheses. Issues associated with the existence and the design of such a triplet will be 
discussed later in Section 48.4. For the time being, we observe that the constraints (Equation 48.24) still 
interpret the ability, of the controller, to generate the feedforward input necessary to keep e(t) = 0 in 
steady state. In steady state, in fact, a controller that solves the problem generates a control of the form 


Uss(t) = A.(xc(t), 0) = H.(1.(w(t)), 0) == v(w(t)), 


which, as predicated by the nonlinear regulator equations, is a control able to force a steady state trajectory 
of the form x(t) = m(w(t)) and consequently to keep e(t) identically zero. 

The nonlinear regulator equations can be given a more tangible form if the model (Equation 48.1) 
of the plant is affine in the input u and, viewed as a system with input u and output e, has a globally 
defined normal form. * This means that, in suitable (globally defined) coordinates, the composition of 
plant (Equation 48.1) and exosystem (Equation 48.2) can be modeled by equations of the form 


w= s(w), 
BSF Wee, S15 058) 
1 = &2 
ee (48.28) 
Ey =&r, 
E, = a(w,z,£1,...,&-) +b(w,z,£1,.--,E us 
e= 6&1 


in which r is the relative degree of the system, z € R”~" and b(w, z, &1,...,&,), the so-called high-frequency 
gain, is nowhere zero. 


If the model of the plant is available in normal form, the nonlinear regulator equations 48.23 can be 


dealt with as follows. Let 1(w) be partitioned, consistently with the partition of the state (z,&1,...,&,) of 
Equation 48.28, into 


m(w) = col(ato(w), 11 (w),..., 1 (w)) 


in which case the equations in question become 


270) = f(w, Xo(w), m1 (w),..., (Ww), 

aver =m41(w) i=1,...,r—-1, 

Ow 

n s(w) = a(w, mo(w), 1(w),...,Ur(w)) + b(w, mo(w), T1(w),...5 Tr(w))W(w), 


0= z\(w). 


From these, we deduce that 


m1 (w) =---=1,(w) =0, 
while mo(w) satisfies 

O70 sw) = folw, ol) (48.29) 

— s(w) = fo(w, mo0(w)), : 

aw 0 0 
in which 

folw, z) =f(w,z,0,...,0). 
Moreover, 


W(w) = — go(w, To(w)), (48.30) 


* See Section 57.1 of Chapter 57 for the definition of relative degree and normal form. 
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in which 
Bis a(w, z,0,...,0) 
w, Z) = ————___.. 
au b(w, z,0,...,0) 
The autonomous system 

w=s(w), 
OH (48.31) 
zZ=fo(w, Z), 


characterizes the so-called zero dynamics of Equation 48.28. Thus, to say that the nonlinear regulator 
equations 48.23 have a solution is to say that the zero dynamics of Equation 48.28 possess an invariant 
manifold expressible as the graph of a map z = 119(w) defined on W. 


48.4 Convergence to the Required Steady State 


Mimicking the design philosophy chosen in the case of linear systems, it is natural to look at a controller 
of the form 


u=y(n) +4, 


ees re (48.32) 


where y(-) and ¢(-) satisfy (Equation 48.27) for some t(-) and where v, v’ are additional controls. If these 
controls vanish in steady state, the graph of the nonlinear map 


weWrh (*) = Ea (48.33) 


is by construction an invariant manifold in the composite system 


w=s(w), 
x=f(w.x, y(n) + v), (48.34) 
A= on) +¥. 


The regulated variable e vanishes on the graph of Equation 48.33. Thus, if the additional controls v and v’ 
are able to steer all trajectories to this invariant manifold, the problem of output regulation is solved. In 
the case of linear systems, the success of a similar design philosophy was made possible by the additional 
assumption that all zeros of the controlled plant had negative real part. In what follows, we show how the 
approach in question can be extended to the case of nonlinear systems. 

For convenience, we begin by addressing the special case in which r = 1 and b = 1, deferring to Section 
48.6 the discussion of more general cases. Picking, as in the case of linear systems, v’ = Gv, system 
(Equation 48.34) reduces to a system of the form 


w= s(w), 

z=f(w,z,&1), 
€) =a(w,z,€1)+ y(n) +», (48.35) 
= 9(n) + Gy, 

e=&, 


which, in what follows, will be referred to as the augmented system. The (compact) set of admissible initial 
conditions of Equation 48.35 is a set of the form W x Z x & x H. 
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System (Equation 48.35) still has relative degree r = 1 between input v and output e, with a normal 
form which can be revealed by simply changing y into 


X=n—- GE, 
which yields 
w= s(w), 
Zz = f(w,z,&1); 
X = (KX + GE1) — Gy(x + GE1) — Ga(w, z, £1), (48.36) 
£) = a(w,z, £1) + v(x + Ge) +¥, 
e= 1. 


Observe, in this respect, that the zero dynamics of this system, which will be referred to as the augmented 
zero-dynamics, obtained by entering the constraint e = 0, are those of the autonomous system 


w=s(w), 
Z = fo(w, z), (48.37) 
X= (x) — Gy(X) — Gaol, z). 


By virtue of Equation 48.27 through 48.30, it is readily seen that the manifold 
M = {(w,z, xX): w € W,z = 1o(w), xX = t(w)} (48.38) 


is an invariant manifold of Equation 48.37. 
It is now convenient to regard system (Equation 48.36) as feedback interconnection of a system with 
input &) and state (w, z, x) and of a system with inputs (w, z, x) and v and state &). In particular, setting 


p=col(w,z, x) 


the system in question can be regarded as a system of the form 


p=M(p)+N(p, £1); 


; 48.39 
&) = H(p)+J(p, €1) + 0p, E1)v, \ 


in which M(p) and H(p) are defined as 


s(w) 


M(p)= ( Sow, z) 
e(X) — Gy(x) — Gqgo(w, z) 


and 
H(p) = go(w,z) + v0), 


while N(p, £1) and J(p, &1) are residual functions satisfying N(p, 0) = 0 and J(p, 0) = 0, and b(p, €)) = 1. 
The advantage of seeing system (Equation 48.36) in this form is that, under appropriate hypotheses, 

the control 
er (48.40) 


keeps all admissible trajectories bounded and forces &1(f) to zero as t — oo. In fact, the following result 
holds (see, e.g., [4,9]). 
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Theorem 48.1: 


Consider a system of the form (Equation 48.39) with v as in Equation 48.40. Suppose that M(p), N(p, &1), 
H(p), J(p,&1), and b(p,&1) are at least locally Lipschitz and b(p, 1) > 0. Let the initial conditions of the 
system range in a compact set P x &. Suppose there exists a set A which is locally exponentially stable for 
p=M(p), with a domain of attraction that contains the set P. Suppose also that H(p) = 0 for all p€ A. 
Then, there is a number k* such that, for all k > k*, the set A x {0} is locally exponentially stable for the 
interconnection (Equations 48.39 through 48.40), with a domain of attraction that contains P x ©. 


Applying this result to system (Equation 48.36), it is observed that the system p = M(p) coincides 
with Equation 48.37, that is with the zero dynamics of the augmented system (Equation 48.36). The 
set (Equation 48.38) is an invariant manifold of these dynamics and, by construction, the map H(p) 
vanishes on this set. Thus, it is concluded that if the set (Equation 48.38) is locally exponentially stable 
for Equation 48.37, with a domain of attraction that contains the set of all admissible initial conditions, 
the choice of a high-gain control as in Equation 48.40 suffices to steer &, to zero and hence to solve the 
problem of output regulation. 

For convenience, we summarize the result obtained so far as follows. 


Corollary 48.1: 


Consider a system in normal form (Equation 48.28) with r = 1 and b = 1. Suppose (Equation 48.29) holds 
for some mo(w). Let y(w) be defined as in Equation 48.30 and p(n) and y(n) be such that (Equation 48.27) 
hold for some t(w). Consider a controller of the form 


1 = o(n) — Gke. 


If the manifold (Equation 48.38) is locally exponentially stable for Equation 48.37, with a domain of 
attraction that contains the set of all admissible initial conditions, there exists k* such that, for all k > k*, 
the positive orbit of the set of admissible initial conditions in bounded and e(t) + 0 as t > oo. 


The main issue that remains to be addressed, in this framework, is to determine whether or not 
the desired asymptotic properties of the invariant manifold (Equation 48.38) of Equation 48.37 can be 
obtained. In this respect, the asymptotic properties of the zero dynamics (Equation 48.31) of the controlled 
plant—on the one hand—and the choice of {¢(-), y(-), G}—on the other hand—indeed play a major role. 
This issue is addressed in the next section. For the time being, we observe that, in the case of linear 
systems, the dynamics of Equation 48.37 reduce to linear dynamics, modeled by 

w= Sw, 

Z= Bow+ Aoz, 

X = (®@— GP) x — G(Dow + Coz), 
in which Ag is a matrix whose eigenvalues coincide with the zeros of Equation 48.19. The maps mo(w) 
and t(w) are linear maps, 19(w) = Tow and t(w) = Tw, satisfying 


ToS =Bo+Aolo, TS=@T, PTT=—(Do+ Colo). 
Changing z, x into Z = z — IIgw and x = x — Tw, respectively, these dynamics can be rewritten as 


w = Sw, 
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Z= Aoz, 
Y = (® — GI) X — GCoz, 


from which it is readily seen, as expected, that ifall zeros of Equation 48.19 and all eigenvalues of (® — GI) 
have negative real part, the dynamics in question have the desired asymptotic properties. 


48.5 The Design of the Internal Model 


System (Equation 48.37) can be interpreted as the cascade of 


w=s(w), 
Z=fo(w, z), (48.42) 
y= 40(w,z), 
and of 
X= OC) — Gy(X) — Gy. (48.43) 
We are interested in finding hypotheses under which there exists a triplet {¢(-), y(-), G} such that Equa- 
tion 48.27, with W(w) = —qo(w, mo(w)), holds for some t(-) and such that the resulting invariant manifold 


(Equation 48.38) is locally exponentially stable, with a domain of attraction that contains the set of all 
admissible initial conditions. The first obvious hypothesis is that the set z= m19(w) is a locally expo- 
nentially stable (invariant) manifold of Equation 48.42, with a domain of attraction that contains the 
set W x Z. This assumption is the nonlinear analogue of the assumption that system (Equation 48.19) 
has all zeros with negative real part and is usually referred to, with some abuse of terminology, as the 
minimum-phase assumption. 

Once this is assumed, the matter is to determine the existence of a triplet {¢(-), y(-), G} with the 
appropriate properties. As we have seen, this is always possible for a linear system. The basic argument 
behind the construction of the pair (®, I) that makes conditions (Equation 48.12) fulfilled (recall that 
these are the linear version of Equation 48.27) is that, by Cayley-Hamilton’s theorem, 


WS? = —(soW + 5) US +--+ +5g_,VsSt}). 


It is seen from this that the function u,,(t) = Yw(t), with w(t) solution of w = Sw, satisfies the linear 
differential equation 
u(t) = —sou,,(t) — su) (t) — --- — sq_ud-Y (8). 

Motivated by this interpretation, we may assume, in the nonlinear case, the existence of an integer 
d, of a (locally Lipschitz) function @: R4 — R with the property that, for any wo € W, the solution 
w(t) of w = s(w) passing through wo at time t = 0 is such that the function uss(t) = W(w(t)), in which 
wW(w) = —go(w, Xo(w)), satisfies the nonlinear differential equation 

u(t) = o(u,,(t), u(t), ..., u'2-Y (2). (48.44) 

If this property is assumed, it is easy to find a pair ¢(-), y(-) such that Equation 48.27 holds for some 

t(-). In fact, it suffices to set 


u(w) = col(y(w), Ley w),..., LE“ Ww), 
to pick any function , : R¢  R which is globally Lipschitz and agrees with (-) on t(W), and set 
N2 
o(n) = ae > y)=n1. (48.45) 
Nd 
oc(m1, Rene Nd)» 
A simple calculation shows that conditions (Equation 48.27) hold. 
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Once the ¢(-), y(-) have been determined, it remains to show that a vector G can be found yielding the 
desired asymptotic properties. To this end, set 


K=x-1tw), Y= qo(w,z) — go(w, To(w)) 


and observe that ; 
X= OX + t(w)) — G(t(w)) — GX — Gy. (48.46) 


Since z = 1o(w) is an asymptotically stable (invariant) manifold of Equation 48.42, the input y of Equa- 
tion 48.46 decays to zero as t > oo. Moreover, by construction, % = 0 is an equilibrium of Equation 48.46 
when y = 0. 

The pair {¢(-), y(-)} defined in Equation 48.45 is uniformly observable (see [5]). Therefore, it is likely 
that the desired asymptotic properties of Equation 48.46 could be achieved by picking G as in the design 
of high-gain observers. Proceeding in this way, choose 


G=D,Go, 
in which 
Dy, = diag(k, Ky... «4), 
Go = col(cq_1, Cgq_-25 +++» C0)» 
with cg_1,...,¢9 coefficients of a Hurwitz polynomial 


PO) Seo ees eg 


It is possible to prove (see, e.g., [6]) that, if « is large enough, system (Equation 48.46) is globally input-to- 
state stable, actually with a linear gain function, and that the equilibrium % = 0—achieved for y = 0—is 
globally exponentially stable. This, coupled with the assumption that z = 119(w) is a locally exponentially 
stable (invariant) manifold of Equation 48.42, proves that Equation 48.38 is a locally exponentially stable 
(invariant) manifold of Equation 48.37, with a domain of attraction that contains the set W x Z x R4. 
This makes the result of Corollary 48.1 applicable. 

We have shown, in this way, that under the assumptions that the controlled plant is minimum phase 
and that the family of all “steady state inputs” us(t) = y(w(t)) obeys a (possibly nonlinear) high-order 
differential equation of the form (Equation 48.44), the problem of output regulation can be solved. One 
may wonder whether these assumptions can be weakened, in particular the assumption of the existence 
of a differential equation of the form (Equation 48.44). There is, in fact, an alternative design strategy, 
in which the assumption in question is not needed. This strategy is based on seeking the fulfillment of 
Equation 48.27 with a p(n) of the form 


o(n) = Fy + Gy(n), 
which entails, for the system (Equation 48.43), a structure of the form 
¥ = Fx -Gy, (48.47) 


with F a Hurwitz matrix. In this case, to say that Equation 48.27 are fulfilled is to say that there exists a 
pair (F, G), with F a Hurwitz matrix and a map y(n) such that 


Ww) = y(t(w)), 


at VweW (48.48) 
s(w) = Fr(w) + Gir(w), 
Ow 


hold for some t(w). The relevant result, in this respect, is that such a triplet always exists, if the dimension 
d of F is sufficiently large and y(n) is allowed to be only continuous (and, thus, possibly not locally 
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Lipschitz). Specifically, note that, regardless of what the dimension of the matrix F is, if the latter is 
Hurwitz, a map t(w) fulfilling the second equation of Equation 48.48 always exists. In fact, if the controlled 
plant is minimum phase, and hence all trajectories of Equation 48.42 with initial conditions in W x Z are 
bounded, and if the matrix F is Hurwitz, also all trajectories of 


w = s(w), 
Zz = fo(w, z), (48.49) 
Xx = Fx — Gqo(w, z) 

for any initial conditions in W x Z x R@ are bounded, and converge to a steady state locus. The latter 


is the graph of a map defined on W, in which z= m9(w) and x = t(w), where t(w), if continuously 
differentiable, satisfies 


Ou 
ay = Ft(w) — Gqo(w, 1o(w)) = Fr(w) + Gi(w). (48.50) 
Thus, the real issue is simply when there exists a map y(n) such that the map t(w) which characterizes 


the steady state locus of Equation 48.49 satisfies the first condition in Equation 48.48. This problem has 
been recently answered in [4], in the following terms. 


Proposition 48.1: 


There is an integer £ > 0 such that, if the eigenvalues of F have real part which is less than — £, there exists 
a unique continuously differentiable t(w) which satisfies (Equation 48.50). Suppose 


d>2dim(w) +42. 


Then for almost all choices (see [4] for details) of a controllable pair (F, G), with F a Hurwitz matrix whose 
eigenvalues have real part which is less than —£, the map t(w) satisfies 


Uw) =w2) > W(w1)=W(w2), Vwi, w2) © Wx W, 


As a consequence, there exists a continuous map y : R4 — R such that the first identity in Equation 48.48 
holds. 


This shows that the existence of a triplet {F, G, y(-)} with the desired properties can always be achieved, 
so long as the integer d is large enough. As a consequence, the design procedure outlined earlier in Corol- 
lary 48.1 is always applicable, so long as the controlled plant satisfies the minimum-phase assumption. 
From the constructive viewpoint, however, it must be observed that the result indicated in Proposition 
48.1 is only an existence result and that the function y(-), whose existence is guaranteed, is only known to 
be continuous. Obtaining continuous differentiability of such y(-) is likely to require further hypotheses.* 


48.6 The Case of Higher Relative Degree 


Consider now the case of a system having relative degree higher than 1, but still assume, for simplicity, 
that b(w,z,&1,...,&,) = 1. In addition, assume that the function f(w, z,&1,...,€,) is independent of 


* Closed-form expressions for y(-) and other relevant constructive aspects are discussed in [7]. 


Nonlinear Output Regulation 48-15 


&5,...,&,. Choose, as in Section 48.4, a control of the form (Equation 48.32) with v’ = Gv. This yields an 
augmented system that can be written in the form 


w= s(w), 
z= f(w,z, CZ), 
z= AZ+ BE,, 
. br (48.51) 
Er = a(w,z,Z,&-) +y(m) +4, 
= o(y) + Gy, 
e= Cz, 
in which Z = col(&,...,&,—1) and 
0 1 0 0 0 
0 los. 0 0 
A=]- - - -- +], B=]-], C= 0 0 .--- 0). 
000 --- 1 
000 --- 0 1 


The idea is to try to transform this system into a system of the form (Equation 48.39) and to try to use 
again, if possible, the result of Theorem 48.1. Transformation into a system of the form (Equation 48.39) 
can be achieved by changing &, into 

0 = &, — Kz, 


and y into 
XY =n - G8, (48.52) 
in which K is a vector of design parameters yet to be determined. In this way, the augmented system 
(Equation 48.51) can be written as 
P=M(p)+N@.9), 


. (48.53) 
8 = H(p) + J(p, 9) + b(p, 8)v, 


in which p = col(w, z, Z, x). The subsystem p = M(p), which coincides with the zero dynamics of Equa- 
tion 48.51 if the latter is viewed as a system with input v and output 9, is a system of the form 


w s(w) 

z)_ fo(w,z) +filw, z, CZ) 

z| (A+ BK)z > (48.54) 
x (x) — Gy(x) — Gqo(w, z) — Gqi(w, z; Z) 


in which 


filw, z, Cz) = f(w, z, CZ) — fo(w, z), 
qi(w, Z,Z) = a(w, z,Z, Kz) — K(A+ BK)z — qo(w, z) 


are functions vanishing at z = 0. The map H(p) is a map of the form 
A(p) = gow, z) + v(x) + qi, z, Z), 
the residual functions N(p, 0) and J(p, 8) vanish at 6 = 0, and b(p, 9) = 1. Observe that the manifold 
M = {(w,z,Z,%): we W,z=To(w), Z =0, x = t(w)} (48.55) 


is invariant for the dynamics of Equation 48.54 and that the map N(p) vanishes on this set. 
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It follows from Theorem 48.1 that, if the manifold (Equation 48.55) is locally exponentially stable for 
Equation 48.54, with a domain of attraction that contains the set of all admissible initial conditions, the 
choice of a high-gain control 

v = —k0 = —k(&, — Kz) 


suffices to steer 0 to zero and p to Equation 48.55. Since &;, a component of Z, is zero in Equation 48.55, 
then & also is steered to zero and the problem of output regulation is solved. 

The control law proposed in this way is a law which presupposes the availability of &1,...,&,, that 
is, of the regulated variable e and its derivatives e“),...,e—)). This is not a problem, however, since 
appropriate substitutes for these variables can be generated, so long as the set of admissible initial condi- 
tions is compact, by means of an appropriate r-dimensional system driven by e, as suggested in [8]. The 
applicability of the methods depends therefore on the ability to choose the design parameters in such a 
way that the manifold (Equation 48.55) is locally exponentially stable for Equation 48.54, with a domain 
of attraction that contains the set of all admissible initial conditions. In this respect, it must be observed 
that the system in question can be seen as a cascade of 


w= s(w), 
= fo(w,z) + Sil, z, CZ), (48.56) 
Z= (A+ BK)z, 


y = qo(w,z) + qi(w, z, Z); 


and of system (Equation 48.43). 

It is known that, if the controlled plant is minimum phase, that is, if the set z = m9(w) is a locally 
exponentially stable (invariant) manifold for the dynamics of Equation 48.42, with a domain of attraction 
that contains the set W x Z, then given any compact set Z, there is a matrix K such that the set (z,Z) = 
(sto(w), 0) is a locally exponentially stable invariant manifold for the dynamics of Equation 48.56, with 
a domain of attraction that contains the set W x Z x Z (see, e.g., [9]). Thus, if K is chosen in this way, 
either one of the two methods for the design of {¢(-), y(-), G} suggested in Section 48.5 can be used to 
complete the design of the regulator. 

We conclude by observing that, if the high-frequency gain on the system is not equal to 1, identical 
results hold, which can be proven using the change of variable 


; 1 
= cf ds, 
M 7 0 b(w, z,&1,..-5€r—1, 5) 


instead of Equation 48.52. On the contrary, the assumption that f(w, z,&1,...;,&,) is independent of 
&,...,&, can only be removed at the expense of other assumptions, such as the property, of system 


w = s(w), 


Zz =f(w,z,&1, tee »&r)s 


of being input-to-state stable, in the input (&1,...,&,). 
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49.1 Introduction 


Lyapunov functions represent the primary tool for the stability analysis of nonlinear systems. They verify 
the stability of a given trajectory, and they also provide an estimate of its region of attraction. The purpose 
of this text is to illustrate the utility of Lyapunov functions in the synthesis of nonlinear control systems. 
We will focus on recursive state-feedback design methods which guarantee robust stability for systems 
with uncertain nonlinearities. Lyapunov design is used in many other contexts, such as dynamic feedback, 
output feedback, gain assignment, estimation, and adaptive control, but such topics are beyond the scope 
of this chapter. 

Given a state-space model of a plant, the Lyapunov design strategy is conceptually straightforward and 
consists of two main steps: 


1. Construct a candidate Lyapunov function V for the closed-loop system. 
2. Construct a controller which renders its derivative V negative for all admissible uncertainties. 


Such a controller design guarantees, by standard Lyapunov theorems, the robust stability of the closed- 
loop system. The difficulty lies in the first step, because only carefully constructed Lyapunov functions 
can lead to success in the second step. In other words, for an arbitrary Lyapunov function candidate V, it 
is likely that no controller can render V negative in the entire region of interest. Those select candidates 
which do lead to success in the second step are called control Lyapunov functions. Our first design step 
should, therefore, be to construct a control Lyapunov function for the given system; this will then insure 
the existence of controllers in the second design step. 

In Section 49.2 we review the Lyapunov redesign method, in which a Lyapunov function is known for 
the nominal system (the system without uncertainties) and is used as the control Lyapunov function for 
the uncertain system. We will see that this method is essentially limited to systems whose uncertainties 
satisfy a restrictive matching condition. In Section 49.3 we show how such limitations can be avoided 
by taking the uncertainty into account while building the control Lyapunov function. We then present 
a recursive robust control design procedure in Section 49.4 for a class of uncertain nonlinear systems. 
Flexibilities in this recursive design are discussed in Section 49.6. 
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49.2 Lyapunov Redesign 


A standard method for achieving robustness to state-space uncertainty is Lyapunov redesign; see [12]. 
In this method, one begins with a Lyapunov function for a nominal closed-loop system and then uses 
this Lyapunov function to construct a controller which guarantees robustness to given uncertainties. To 
illustrate this method, we consider the system, 


x = F(x) + G(x)u+ A(x, 0), (49.1) 


where F and G are known functions comprising the nominal system and A is an uncertain function known 
only to lie within some bounds. For example, we may know a function p(x) so that | A(x, f) |< p(x). A 
more general uncertainty A would also depend on the control variable u, but for simplicity we do not 
consider such uncertainty here. We assume that the nominal system is stabilizable, that is, that some state 
feedback unom(x) exists so that the nominal closed-loop system, 


x = F(x) + G(x)unom(x), (49.2) 


has a globally asymptotically stable equilibrium at x = 0. We also assume knowledge of a Lyapunov 
function V for this system so that 


VV (x) [F(x) + G(x) Unom(x)] < 0 (49.3) 


whenever x # 0. Our task is to design an additional robustifying feedback u, p(x) so that the composite 
feedback u = Unom + Uyop robustly stabilizes the system (Equation 49.1), that is, guarantees stability for 
every admissible uncertainty A. It suffices that the derivative of V along closed-loop trajectories is negative 
for all such uncertainties. We compute this derivative as follows: 


V = VV (x) [F(x) + G(x)Unom(x)] + VV (x) [G(x)Urop(x) + A(x, )] (49.4) 


Can we make this derivative negative by some choice of u,o,(x)? Recall from Equation 49.3 that the first 
of the two terms in Equation 49.4 is negative; it remains to examine the second of these terms. For those 
values of x for which the coefficient V V(x) - G(x) of the control u, p(x) is nonzero, we can always choose 
the value of u,.p(x) large enough to overcome any finite bound on the uncertainty A and thus make 
the second term in Equation 49.4 negative. The only problems occur on the set where VV(x) - G(x) = 0, 
because on this set 

V =VV(x)- F(x) + VV(x)- A(x, t) (49.5) 


regardless of our choice for the control. Thus to guarantee the negativity of V, the uncertainty A must 
satisfy 
VV(x)- F(x) + VV(x)- A(x, t) < 0 (49.6) 


at all points where V V(x) - G(x) = 0. This inequality constraint on the uncertainty A is necessary for the 
Lyapunov redesign method to succeed. Unfortunately, there are two undesirable aspects of this necessary 
condition. First, the allowable size of the uncertainty A is dictated by F and V and can thus be severely 
restricted. Second, this inequality (Equation 49.6) cannot be checked a priori on the system (Equation 49.1) 
because it depends on the choice for V. 

These considerations lead to the following question. Are there structural conditions that can be imposed 
on the uncertainty A so that the necessary condition (Equation 49.6) is automatically satisfied? One such 
structural condition is obvious. If we require that the uncertainty A is of the form, 


A(x, t) = G(x). A(x, t), (49.7) 


for some uncertain function A, then clearly VV - A =0 at all points where VV -G=0, and thus the 
necessary condition (Equation 49.6) is satisfied. In the literature, Equation 49.7 is called the matching 
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condition because it allows the system (Equation 49.1) to be written 
x = F(x) + G(x)[u+ A(x, £)] (49.8) 


where now the uncertainty A is matched with the control u, that is, it enters the system through the same 
channel as the control [2,4,12]. 

There are many methods available for the design of u;oy(x) when the matching condition (Equa- 
tion 49.7) is satisfied. For example, if the uncertainty A is such that | A(x, t) |< p(x) for some known 
function 9, then the control, 

(VV(x)- G(x)" 
| VV(@)- GO) |” 


Uyob (x) = (x) (49.9) 


yields 
V < VV (x) [F(x) + G(x) Unom(x)] + | VV (x) - G(x) | [-(x)+ | AG, #) |] (49.10) 


The first term in Equation 49.10 is negative from the nominal design (Equation 49.3), and the second 
term is also negative because we know that | A(x, t) |< 6(x). The composite control u = unom + Uyop thus 
guarantees stability and robustness to the uncertainty A. This controller (Equation 49.9), proposed, for 
example, by [8], is likely to be discontinuous at points where V V(x) - G(x) = 0. Indeed, in the scalar input 
case, Equation 49.9 becomes 

Urop(X) = —P(x)sgn (Vv V(x): G(x)) (49.11) 
which is discontinuous unless 0(x) = 0 whenever V V(x) - G(x) = 0. Corless and Leitmann [4] introduced 
a continuous approximation to this controller which guarantees convergence, not to the point x = 0, but 
to an arbitrarily small prescribed neighborhood of this point. We will return to this continuity issue in 
Section 49.5. 

We have seen that, because of the necessary condition (Equation 49.6), the Lyapunov redesign method 
is essentially limited to systems whose uncertainties satisfy the restrictive matching condition. In the next 
sections, we will take a different look at Equation 49.6 and obtain much weaker structural conditions on 
the uncertainty, which still allow a systematic robust controller design. 


49.3 Beyond Lyapunov Redesign 


In the previous section, we have seen that, if a Lyapunov function V is to guarantee robustness to an 
uncertainty A, then the inequality 


VV(x)- F(x) + VV(x)- A(x, t) < 0 (49.12) 


must be satisfied at all points where V V(x) - G(x) = 0. In the Lyapunov redesign method, this inequality 
was viewed as a constraint on the uncertainty A. Now let us instead view this inequality as a constraint on 
the Lyapunov function V. This new look at Equation 49.12 will lead us beyond Lyapunov redesign: our 
construction of V will be based on Equation 49.12 rather than on the nominal system. In other words, we 
will take the uncertainty A into account during the construction of V itself. 

To illustrate our departure from Lyapunov redesign, consider the second-order, single-input uncertain 
system, 


xy = x2 + Aj (x, t) (49.13) 
x2 =u+t A(x, t) (49.14) 


where A; and A) are uncertain functions which satisfy some known bounds. Let us try Lyapunov 
redesign. The first step would be to find a state feedback unom(x) so that the nominal closed-loop system, 


x1 = x2 (49.15) 
x2 = Unom(x) (49.16) 
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has a globally asymptotically stable equilibrium at x = 0. Because the nominal system is linear, this step 
can be accomplished with a linear control law uyom(x) = Kx, and we can obtain a quadratic Lyapunov 
function V(x) =x! Px for the stable nominal closed-loop system. In this case, the necessary condition 
Equation 49.12 becomes 


x2 + Ai(x, f) 
2x™P <0 49.17 
. Ag(x,t) ] 7 et) 
at all points where 2x! P[O 1]! =0, that is, where x. = —cx, for some constant c > 0. We substitute 
x2 = —cx) in Equation 49.17, and, after some algebra, we obtain 

x Ay (x1, —cx1, t) < ext (49.18) 


for all x,t € R. Now suppose our knowledge of the uncertainty A; (x1, —cx1, t) consists of a bound p1 (x) 
so that | Ai(x1, —cx1, t) |< pi(x1). Then Equation 49.18 implies that p1(x1) <c | x1 |, that is, that the 
uncertainty A, is restricted to exhibit only linear growth in x, at a rate determined by the constant c. 
In other words, if the uncertainty A; does not satisfy this c-linear growth, then this particular Lyapunov 
redesign fails. This was to be expected because the uncertainty A, does not satisfy the matching condition. 
The above Lyapunov redesign failed because it was based on the linear nominal system which suggested 
a quadratic Lyapunov function V. Let us now ignore the nominal system and base our search for V directly 
on the inequality (Equation 49.12). Let j1(x;) be a smooth function so that j4(0) = 0, and consider the 
Lyapunov function 
V(x) = x? + [x2 — w(x1)]*. (49.19) 


This function V is smooth, positive definite, and radially unbounded and thus qualifies as a candidate 
Lyapunov function for our system (Equations 49.13 and 49.14). We will justify this choice for V in the 
next section; our goal here is to illustrate how we can use our freedom in the choice for the function wt to 
derive a necessary condition on the uncertainty A; which is much less restrictive than Equation 49.18. 
For V in Equation 49.19, VV(x)- G(x) = 0 if, and only if, x2 = |1(x1), so that the necessary condition 
Equation 49.12 becomes 
X1W(x1) + x1 Ay (x1, U(%1), f) < 0 (49.20) 


for all x1, t € R. Because we have left the choice for js open, this inequality can be viewed as a constraint 
on the choice of V (through |) rather than a constraint on the uncertainty A;. We need only impose a 
structural condition on A; which guarantees the existence of a suitable function j1. An example of such a 
condition would be the knowledge of a bound pj (x1) so that | Ay (x1, x2, t) |< 91 (x1); then Equation 49.20 
becomes 


x1 (x1)+ | x1 | p1(x1) <0 (49.21) 


for all x; € R. It is then clear that we can satisfy Equation 49.21 by choosing, for example, 


(x1) = —x1 — 01(x1)sgn(x1). (49.22) 


A technical detail is that this jt is not smooth at x; = 0 unless p;(0) = 0, which means V in Equa- 
tion 49.19 may not strictly qualify as a Lyapunov function. As we will show in Section 49.5, however, 
smooth approximations always exist that will end up guaranteeing convergence to a neighborhood of 
x = 0in the final design. What is important is that this design succeeds for any function 9; (x1), regardless 
of its growth. Thus the c-linear growth condition on A; which appeared in the above Lyapunov redesign 
through Equation 49.18 is gone; this new design allows arbitrary growth (in x;) of the uncertainty Aj. 

We have not yet specified the controller design; rather, we have shown how the limitations of Lyapunov 
redesign can be overcome through a reinterpretation of the necessary condition (Equation 49.12) as a 
constraint on the choice of V. Let us now return to the controller design problem and motivate our choice 
of V in Equation 49.19. 
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49.4 Recursive Lyapunov Design 


Let us consider again the system (Equations 49.13 and 49.14): 


x1 = x2 + Ay (x, f) (49.23) 
x2 =u+ A(x, t) (49.24) 


We assume knowledge of two bounding functions p;(x,) and p2(x) so that all admissible uncertainties 
are characterized by the inequalities, 


| A(x, t) | < 01(%1) (49.25) 
| Az(x, t) | < p2(x) (49.26) 


for all x € R? and all t € R. Note that the bound p, on the uncertainty A is allowed to depend only on the 
state x); this is the structural condition suggested in the previous section and will be characterized more 
completely below. We will take a recursive approach to the design of a robust controller for this system. 
This approach is based on the integrator backstepping technique developed by [11] for the adaptive control 
of nonlinear systems. The first step in this approach is to consider the scalar system, 


xy = u+Ai(x1, 4%, 6), (49.27) 


which we obtain by treating the state variable x. in Equation 49.23 as a control variable u. This new 
system (Equation 49.27) is only conceptual; its relationship to the actual system (Equations 49.23-49.24) 
will be explored later. Let us next design a robust controller u = (x1) for this conceptual system. By 
construction, this new system satisfies the matching condition, and so we may use the Lyapunov redesign 
method to construct the feedback u = \t(x;). The nominal system is simply x; = u which can be stabilized 
by a nominal feedback tjom = —x,. A suitable Lyapunov function for the nominal closed-loop system 
x, = —x, would be V}(x1) = coe We then choose # = Unom + Uyop, Where U;yop is given, for example, by 
Equation 49.9 with p = 9. The resulting feedback function for u is 


w(x1) = —x1 — 01(x1)sgn(x1). (49.28) 


If we now apply the feedback i = (x) to the conceptual system (Equation 49.27), we achieve V, < —2x? 
and thus guarantee stability for every admissible uncertainty A. Let us assume for now that this function 
w is (sufficiently) smooth; we will return to the question of smoothness in Section 49.5. 

The idea behind the backstepping approach is to use the conceptual controller (Equation 49.28) in 
constructing a control Lyapunov function V for the actual system (Equations 49.23-49.24). If we treat the 
Lyapunov function as a penalty function, it is reasonable to include in V a term penalizing the difference 
between the state x2 and the conceptual control u designed for the conceptual system (Equation 49.27): 


V(x) = Vile1) + Peo — wx). (49.29) 


We have seen in the previous section that this choice satisfies the necessary condition (Equation 49.12) 
and is thus a good candidate for our system (Equations 49.23-49.24). It is no coincidence that we arrived 
at the same expression for 1 in Equations 49.22 and 49.28 both from the viewpoint of the necessary 
condition (Equation 49.12) and from the viewpoint of the conceptual system (Equation 49.27). 

Let us now verify that the choice of Equation 49.29 for V leads to a robust controller design for the 
system (Equations 49.23-49.24). Computing V we obtain 


V = 2x\ [2 + Ai(x, t)]4 2b — wr)] [u+ Ao(x, t) — pw! (x1)[2 + Ar(x, 1] (49.30) 
where 1’ := dy /dx. Rearranging terms and using Equation 49.28, 
V <—2xq + 2[x2 — w(xr)] [a tut Ao(x, 1) — wer) 2 + Ar, 6] (49.31) 


The effect of backstepping is that the uncertainties enter the expression (Equation 49.31) with the same 
coefficient as the control variable u; in other words, the uncertainties effectively satisfy the matching 
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condition. As a result, we can again apply the Lyapunov redesign technique; the feedback control u = 
Unom + Urob With a nominal control, 


Unom(x) = — [x2 — W(x1)] — x4 + (x1) x25 (49.32) 
yields 
V < —2xq — 2 [x2 — we)? +2 [x2 — 1021) [Mrop + Art) — w'(a)Ar(, 1). (49.33) 
We may complete the design by choosing u,op as in Equation 49.9: 
Uroo(x) = —P(x)sgn [x2 — 1 (%1)] (49.34) 
where p is some function satisfying | Az(x, t) — (x1) A1(x, t) |< p(x). This yields 
V < —2xt — 2 be — wm)? (49.35) 


for all admissible uncertainties A; and A2, and thus robust stability is guaranteed. 


Example 49.1: 


The above second-order design applies to the following system: 


xX, =X2+ Ar(x, t) (49.36) 
x2 =u (49.37) 


where we let the uncertainty A; be any function satisfying | A7(x,t) |< |x7|?. In this case, the 
function w in Equation 49.28 would be (x1) = —x1 _x3 which is smooth as required. The nom- 
inal control Unom is given by Equation 49.32, which, for this example, becomes 


Uinom(x) = — [x2 + x1 +8] <4 = (14 3x9 x2 (49.38) 
Adding the robustifying term (Equation 49.34), we obtain the following robust controller: 
a 2 2 2 5 3 
u(x) = — [x2 +xq +x7] —X1—- (1 + 3x?) x2 - ( xX, |? +3 | x7 | ) sgn (x +X +43) é (49.39) 


This robust controller is not continuous at points where x7 + x; + oe = 0;an alternate smooth design 
will be proposed in Section 49.5. 

The above controller design for the system (Equations 49.23 and 49.24) is a two-step design. In the 
first step, we considered the scalar system (Equation 49.27) and designed a controller u = \1(x,) which 
guaranteed robust stability. In the second step, we used the Lyapunov function (Equation 49 29) to 
construct a controller (Equations 49.32 + 49.34) for the actual system (Equations 49.23 and 49.24). 
This step-by-step design can be repeated to obtain controllers for higher order systems. For example, 
suppose that instead of the system (Equations 49 23 and 49.24), we have the system 


x1 =x2 + Ar (x,t) (49.40) 
Xo =Z+ Adr\x,t) 
Z=v+A3(x,z, t) (49.41) 


where A, and A> areas in Equations 49 25 and 49.26, A3 is anew uncertainty, and v is the control 
variable. We can use the controller u(x) := Unom(x) + Ugh (x) designed above in Equations 49.32 + 
49.34 to obtain the following Lyapunov function W for our new system: 


W(x,z) = Vix) + [z— u(x)? (49.42) 


Here V is the Lyapunov function (Equation 49 29) used above, and we have simply added a term 
which penalizes the difference between the state variable z and the controller u(x) designed above 
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for the old system (Equations 49 23 and 49.24). If u(x) is smooth, then Equation 49.42 qualifies as a 
candidate Lyapunov function for our new system; see Section 49.5 below for details on choosing a 
smooth function u(x). We can now construct a controller v = u(x, z) for our new system in the same 
manner as the construction of \1(x;) and u(x) above. 

We have thus obtained a systematic method for constructing Lyapunov functions and robust 
controllers for systems of the form, 


X1 =X2+ Ar\x,t), (49.43) 
X2 = x3 + Az(x, t), 


Xn =ut+ An(x, t). (49.44) 


The Lyapunov function will be of the form 
n-1 ; 
Vix) =x7 + > Dig — wiles. ))] (49.45) 
i=1 


where the functions 1; are constructed according to the recursive procedure described above. For 
this approach to succeed, it is sufficient that the uncertainties Aj; satisfy the following strict feedback 
condition: 


| Aj(x, t) |S pj %1,...X7) (49.46) 


for known functions p;. The restriction here is that the ith bound p; can depend only on the first i 
States (x1,...,X;). 

This recursive Lyapunov design technique applies to uncertain systems more general than 
Equations 49.43-49.44. Multi-input versions are possible, and the strict feedback condition (Equa- 
tion 49.46) can be relaxed to allow the bound 9; to depend also on the state x;,;. In particular, the 
bound pr on the last uncertainty Ap can also depend on the control variable u. More details can be 
found in [5,14,16,17]. 


49.5 Smooth Control Laws 


The control law 1; designed in the ith step of the recursive design becomes part of the Lyapunov function 
(Equation 49.45) in the next step. It is imperative, therefore, that each such function \1; be differentiable. 
To illustrate how smooth functions can be obtained at each step, let us return to the second-order design 
in Section 49.4. The first step was to design a robust controller u = (x1) for the conceptual system, 


x) =u+ Aj (x1, 4, t) (49.47) 


with | Ay |< p1(x1). In general, when pe; (0) 4 0, we cannot choose 1 as in Equation 49.28 because of the 
discontinuity at x; = 0. One alternative is to approximate Equation 49.28 by smooth function as follows: 


x] 
w(x}) = —x] pi) Gq) (49.48) 


where 8(x;) is a smooth, strictly positive function. This choice for 1 is once differentiable, and it reduces 
to the discontinuous function (Equation 49.28) when 8 = 0. We compute the derivative of Vj(x1) = ae 
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for the system (Equation 49.47) with the smooth feedback (Equation 49.48): 


2 


V < —2xt — 2p) (%1) ——4 — + 291 (x1) | x1 | 
: | x1 | +8(x1) 
8(x1) | x1 | 
< —2xf + 261 (x1) —_—_— 
; | x1 | +8(o1) 
< —2x? + 201(x1)8(x1) (49.49) 


If we now choose 8(x;) so that p18 is small, we see that V) <0 except in a small neighborhood of x; = 0. 
In the second design step, we will choose Unom as before in Equation 49.32, but instead of Equation 49.33 
we obtain the Lyapunov derivative, 


V < —2xf + 21 (x1)8(x1) — 2 [x2 — war)? + 2 [x2 — W(o1)] [ron + Ar x, t) — (a) Ar(x, #)] (49.50) 


where the extra term 2:18 comes from Equation 49.49 and is a result of our smooth choice for |1(x;). Our 
remaining task is to construct the robustifying term u; 9. Using the bound | A2(x, t) — (x1) Ai(%, t) |< 
p(x) as before, we obtain 


V < —2x} + 21 (x1)8(x1) — 2 [x2 — wor)? +2 [x2 — (x1)] urop + 2 Lx — WO) BCX) (49.51) 


We cannot choose uy) as before in Equation 49.34 because it is not continuous at points where x2 = (L(x). 
We could choose a smooth approximation to Equation 49.34, as we did above for the function |1, but, to 
illustrate an alternative approach, we will instead make use of Young’s inequality, 


1 
2ab < —a* +8b* (49.52) 
E 


which holds for any a, b € R and any ¢ > 0. Applying this inequality to the last term in Equation 49.51, 
we obtain 


; 1 _ 
V < —2xf + 291(x1)8(1) E Z| [x2 — wor]? +2 [x2 — w(x1)] trop + 2(x) [0(x)]? (49.53) 
where (x) is a smooth, strictly positive function to be chosen below. Thus 


Urob(X) = 5 [x2 — w(x))] (49.54) 


is smooth and yields 


VS ~2xq — 2 [x2 — wear]? + 201 (x1)8(1) + ex) [Cx]? (49.55) 
It is always possible to choose 8(x;) and e(x) so that the right-hand side of Equation 49.55 is negative, 


except possibly in a neighborhood of x = 0. Thus we have gained smoothness in the control law but lost 
exact convergence of the state to zero. 


49.6 Design Flexibilities 


The degrees of freedom in the recursive Lyapunov design procedure outlined above are numerous and 
allow for the careful shaping of the closed-loop performance. However, this procedure is new, and 
guidelines for exploiting design flexibilities are only beginning to appear. Our purpose in this section is 
to point out several of these degrees of freedom and discuss the consequences of various design choices. 
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We have already seen that the choices for the functions |1; in the Lyapunov function (Equation 49.45) 
are by no means unique, nor is it the final choice for the control law. For example, when choosing 
a robustifying term u; , in some step of the design, should we choose a smooth approximation to 
Equation 49.9 as in Equation 49.48, or should we use Young’s inequality and choose Equation 49.54? 
Also, how should we choose the nominal term Unom? Is it always good to cancel nonlinearities and apply 
linearlike feedback as in Equation 49.32, and if so, which gain should we use? Such questions are difficult 
in general, but there are some guidelines that apply in many situations. For example, consider the task of 
robustly stabilizing the point x = 0 of the simple scalar system, 


x= —x? +u+ A(x, t) (49.56) 


where A is an uncertain function with a known bound | A(x, f) |<| x |. Because the matching condition 
is satisfied, we can apply the Lyapunov redesign method and choose u = ungm + Uyop. One choice for the 
nominal control would be ujom(x) = x? — x, which, together with a robustifying term as in Equation 49.9 
yields a control law 

u(x) =x? — 2x (49.57) 


This control law is valid from the viewpoint of Lyapunov redesign, and it indeed guarantees robustness to 
the uncertainty A. In such a choice, however, large positive feedback x? is used to cancel the nonlinearity 
—x? in Equation 49.56. This is absurd because the nonlinearity —x? is beneficial for stabilization, and 
positive feedback x? will lead to unnecessarily large control effort and will cause robustness problems more 
severe than those caused by the uncertainty A. Clearly, a much more reasonable choice is u(x) = —2x, 
but how can we identify better choices in a more general setting? One option would be to choose the 
control to minimize some meaningful cost functional. For example, the control 


u(x) =x? —x—xV x4 — 2x2 +2 (49.58) 


minimizes the worst-case cost functional 


CO 
on / [x2 + 2] at (49.59) 
|Als|x| 70 


for this system Equation 49.56. The two control laws (Equations 49.57 and 49.58) are shown in Figure 49.1. 
We see that the optimal control (Equation 49.58) recognizes the benefit of the nonlinearity —x? in 
Equation 49.56 and accordingly produces little control effort for large x. Moreover, this optimal control is 
never positive feedback. Unfortunately, the computation of the optimal control (Equation 49.58) requires 
the solution of a Hamilton-Jacobi-Isaacs partial differential equation, and will be difficult and expensive 
for all but the simplest problems. 

As a compromise between the benefits of optimality and its computational burden, we might consider 
the inverse optimal control problem, summarized for example by [7]. In this approach, we start with 
a Lyapunov function as in the Lyapunov redesign method above. We then show that this Lyapunov 
function is in fact the value function for some meaningful optimal stabilization problem, and we use this 
information to compute the corresponding optimal control. Freeman and Kokotovic¢ [6] have shown that 
the pointwise solutions of static minimization problems of the form, 


minimize u’Su, S=S' >0, (49.60) 
subject to sup [V(x, u, A)+ o(x)] <0, (49.61) 
A 


yield optimal controllers (in this inverse sense) for meaningful cost functionals, where V is a given 
control Lyapunov function and o is a positive function whose choice represents a degree of freedom. 
When the system is jointly affine in the control u and the uncertainty A, this optimization problem 
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FIGURE 49.1 Comparison of the control laws Equation 49.57 (solid) and Equation 49.58 (dotted). 


(Equations 49.60 and 49.61) is a quadratic program with linear constraints and thus has an explicit 
solution u(x). For example, the solution to Equations 49.60 and 49.61 for the system (Equation 49.56) 
with V = x? and o = 2x? yields the control law, 


x3—2x when x? <2, 


0 when x2 >2. yao") 


u(x) = | 


As shown in Figure 49.2, this control (Equation 49.62) is qualitatively the same as the optimal control 
(Equation 49.58); both recognize the benefit of the nonlinearity —x? and neither one is ever positive 
feedback. The advantage of the inverse optimal approach is that the controller computation is simple 
once a control Lyapunov function is known. 


FIGURE 49.2 A comparison of the control laws Equation 49.62 (solid) and Equation 49.58 (dotted). 
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We thus have some guidelines for choosing the intermediate control laws 1; at each step of the design, 
namely, we can avoid the wasteful cancellations of beneficial nonlinearities. However, these guidelines, 
when used at early steps of the recursive design, have not yet been proved beneficial for the final design. 

We have shown that the choices for the functions j1; in Equation 49.45 represent important degrees 
of freedom in the recursive Lyapunov design procedure. Perhaps even more important is the choice of 
the form of V itself. Recall that, given a Lyapunov function V; and a control 1; at the ith design step, we 
constructed the new Lyapunov function V;+, as follows: 


Vigilx,-.->%i¢1) = Viler,...5%i) + [xig1 — Wile... x). (49.63) 


This choice for Vj is not the only choice that will lead to a successful design, and we are thus confronted 
with another degree of freedom in the design procedure. For example, instead of Equation 49.63, we can 
choose 

Vi4i(%1, oo »Xi41) =k [Vj(x1,...,xi)] + [X41 — Wi(xX1,... xl . (49.64) 


where k : Ry — Ry, is any smooth, positive-definite, unbounded function whose derivative is everywhere 
strictly positive. This function « represents a nonlinear weighting on the term V; and can have a large 
effect on the control laws obtainable in future steps. 

The last degree of freedom we wish to discuss involves the quadratic-like term in Equations 49.63 and 
49.64. Praly et al. [15] have shown that Equation 49.64 can be replaced by the more general expression 

Xi+1 
Virals oti) =e (Vile ad) + f (x1,...,X;j, 5) ds (49.65) 
Wi (1 ,-- +»%) 

for a suitable choice of the function ¢. Indeed, Equation 49.64 is a special case of Equation 49.65 for 
= 2[s — pj(x1,...,x;)]. This degree of freedom in the choice of ¢ can be significant; for example, it 
allowed extensions of the recursive design to the nonsmooth case by [15]. It can also be used to reduce 
the unnecessarily large control gains often caused by the quadratic term in Equation 49.63. To illustrate 
this last point, let us return to the second-order example (Equations 49.36 and 49.37) given by 


x1 = xX. + A(x, t) (49.66) 
kn =U (49.67) 
where the uncertainty A; is any function satisfying | Ay(x,t) | < | x1 |? . Recall that using the Lyapunov 


function, : 
V(x) =x7 + [x2 +2147] (49.68) 
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FIGURE 49.3 Quadratic-like Lyapunov function (Equation 49.68). (From Freeman, R. A. and Kokotovié, P. V., 
Automatica, 29(6), 1425-1437, 1993. With permission.) 
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FIGURE 49.4 Controller from quadratic-like Lyapunov function (Equation 49.68). (From Freeman, R. A. and 
Kokotovic¢, P. V., Automatica 29(6), 1425-1437, 1993. With permission.) 


we designed the following robust controller (Equation 49.39) for this system: 
u(x) = — [x2 +x1 +x} ] —x,- (1 + 3x7) x2 — (lil? + 3|x1|°) sgn (x2 +x +x?) : (49.69) 


This controller is not continuous at points where x2 +x; + ee = 0. In other words, the local controller 
gain Ou/Ox is infinite at such points. Such infinite gain will cause high-amplitude chattering along the 
manifold M described by x2 + x1 +x} =0. As a result of such chattering, this control law may not be 
implementable because of the unreasonable demands on the actuator. However, as was shown in Sec- 
tion 49.5, we can use this same Lyapunov function (Equation 49.68) to design a smooth robust controller 
u for our system. Will such a smooth controller eliminate the chattering caused by the discontinuity 
in Equation 49.69? Surprisingly, the answer is no. One can show that the local controller gain 0u/Ox2, 
although finite because of the smoothness of i, grows like x° along the manifold M. Thus for large sig- 
nals, this local gain is extremely large and can cause chattering as if it were infinite. Figure 49.3 shows 
the Lyapunov function V in Equation 49.68, plotted as a function of the two variables, z; := x; and 
Z = X_ +x; +x}. A smooth control law a designed using this V is shown in Figure 49.4, again plotted as 
a function of z; and z). Note that the x® growth of the local gain of 7 along the manifold z, = 0 is clearly 
visible in this figure. One might conclude that such high gain is unavoidable for this particular system. 
This conclusion is false, however, because the x° growth of the local gain is an artifact of the quadratic 
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FIGURE 49.5 New flattened Lyapunov function. (From Freeman, R. A. and Kokotovi¢, P. V., Automatica, 29(6), 
1425-1437, 1993. With permission.) 
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FIGURE 49.6 Controller from flattened Lyapunov function. (From Freeman, R. A. and Kokotovic, P. V., Automat- 
ica, 29(6), 1425-1437, 1993. With permission.) 


form of the Lyapunov function (Equation 49.68) and has nothing to do with robust stability requirements 
for this system. Freeman and Kokotovic¢ [6] have shown how to choose the function ¢ in Equation 49.65 
to reduce greatly the growth of the local gain. For this example (Equations 49.66 and 49.67), they achieved 
a growth of x7 as opposed to the x® growth caused by the quadratic V in Equation 49.68. Their new, 
flattened Lyapunov function is shown in Figure 49.5, and the corresponding control law is shown in Fig- 
ure 49.6. The x? versus x? growth of the local gain is evident when comparing Figures 49.6 and 49.4. The 
control signals generated from a particular initial condition are compared in Figure 49.7. These controls 
produce virtually the same state trajectories, but the chattering caused by the old control law has been 
eliminated by the new one. 


u(old) 


u(new) 


Time 


FIGURE 49.7 Comparison of control signals. (From Freeman, R. A. and Kokotovi¢, P. V., Automatica, 29(6), 
1425-1437, 1993. With permission.) 
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Further Reading 


Contributions to the development of the Lyapunov design methodology for systems with no uncertainties 
include [1,9,10,13,18]. A good introduction to Lyapunov redesign can be found [12]. Corless [3] has 
recently surveyed various methods for the design of robust controllers for nonlinear systems using 
quadratic Lyapunov functions. Details of the recursive design presented in Section 49.4 can be found in [6, 
14,16]. The state-space techniques discussed in this chapter can be combined with nonlinear input/output 
techniques to obtain more advanced designs (see Chapter 44). Finally, when the uncertain nonlinearities 
are given by constant parameters multiplying known nonlinearities, adaptive control techniques apply 
(see Chapter 53). 
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50.1 Introduction and Background 


This chapter investigates variable structure control (VSC) as a high-speed switched feedback control 
resulting in a sliding mode. For example, the gains in each feedback path switch between two values 
according to a rule that depends on the value of the state at each time instant. The purpose of the 
switching control law is to drive the plant’s state trajectory onto a prespecified (user-chosen) surface in 


the state space and to maintain the plant’s state trajectory on this surface for all subsequent time. This 


surface is called a switching surface and the resulting motion of the state trajectory a sliding mode. When 
the plant state trajectory is “above” the surface, a feedback path has one gain and a different gain if the 
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trajectory drops “below” the surface. This surface defines the rule for proper switching. The surface is also 
called a sliding surface (sliding manifold) because, ideally speaking, once intercepted, the switched control 
maintains the plant’s state trajectory on the surface for all subsequent time and the plant’s state trajectory 
then slides along this surface. The plant dynamics restricted to this surface represent the controlled 
system’s behavior. The first critical phase of a VSC design is to properly construct a switching surface, 
so that the plant, restricted to the surface, has desired dynamics, such as stability to the origin, tracking, 
regulation, and so on. 

In summary, a VSC control design generally breaks down into two phases. The first phase is to design 
or choose a sliding manifold/switching surface, so that the plant state restricted to the surface has desired 
dynamics. The second phase is to design a switched control that will drive the plant state to the switching 
surface and maintain it on the surface upon interception. A Lyapunov approach is used in this chapter 
to characterize this second design phase. Here, a generalized Lyapunov function, which characterizes the 
motion of the state trajectory to the surface, is specified in terms of the surface. For each chosen switched 
control structure, one chooses the “gains,” so that the derivative of this Lyapunov function is negative 
definite with respect to the sliding surface, thus guaranteeing motion of the state trajectory to the surface. 

As an introductory example, consider the first-order system x(t) = u(x, t) with control 


-1, ifx>0, 
u(x, t) = —sgn(x) = 
+1, ifx<0. 
Hence, the system with control satisfies x = —sgn(x) with trajectories plotted in Figure 50.1a. Here 


the control u(x,t) switches, changing its value between +1 around the surface o(x, ft) = x = 0. Hence, 
for any initial condition xo, a finite time f, exists for which x(t) = 0 for all t > t), Now, suppose x(t) = 
u(x, t) + v(t), where again u(x, t) = —sgn(x) and v(t) is a bounded disturbance for which sup, |v(t)| < 1. 
As before, the control u(x,t) switches its value between +1 around the surface o(x, t) = x = 0. It follows 
that if x(t) >0, then x(t) = —sgn[x(t)] + v(t) < 0, forcing motion toward the line o(x, t) = x = 0, and if 
x(t) < 0, then x(t) = —sgn[x(t)]+ v(t) > 0, again forcing motion toward the line o(x, t) =x =0. Fora 
positive initial condition, this is illustrated in Figure 50.1b. The rate of convergence to the line depends 
on the disturbance. Nevertheless, a finite time t; exists for which x(t) = 0 for all t > t). If the disturbance 


(a) 


x9 >O0 
State trajectories 


(b) 


xy >0 


State trajectory 


FIGURE 50.1 (a) State trajectories for the system x = —sgn(x); (b) State trajectory for the system x(t) = 
—sgn[x(t)] + v(t). 
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magnitude exceeds 1, then the gain can always be adjusted to compensate for the change. Hence, this VSC 
law is robust in the face of bounded disturbances, illustrating the simplicity and advantage of the VSC 
technique. 

From the above example, one can see that VSC can provide a robust means of controlling (nonlinear) 
plants with disturbances and parameter uncertainties. Further, the advances in computer technology 
and high-speed switching circuitry have made the practical implementation of VSC quite feasible and of 
increasing interest. Indeed, pulse-width modulation control and switched dc-to-dc power converters [1] 
can be viewed in a VSC framework. 


50.2 System Model, Control Structure, and Sliding Modes 
50.2.1 System Model 


The class of systems investigated herein has a state model nonlinear in the state vector x(-) and linear in 
the control vector u(-) of the form 


x(t) = F(x, t, u) = f(x, t) + B(x, t)u(x, t), (50.1) 


where x(t) € R", u(t) € R™, and B(x, t) € R”*”™; further, each entry in f (x,t) and B(x,t) is assumed contin- 
uous with a bounded continuous derivative with respect to x. In the linear time-invariant case, Equation 
50.1 becomes 

x= Ax+Bu (50.2) 


with A n x nand Bn x m being constant matrices. 

As mentioned in the previous section, the first phase of VSC or sliding mode control (SMC) design 
is to choose a manifold S C R", so that the control goal is reached once the state is maintained on S. As 
such we formally define the (n — m)-dimensional switching surface (also called a discontinuity, sliding 
manifold, or equilibrium manifold), as (the possibly time-varying) 


S= {x € R"o(x, t) =o} =() {x € R"o;(x, = 0}; (50.3) 


i=1 


where 
o(x, t) = [o1(x, t),...,0m(x, D]' =0. (50.4) 


(We will often refer to S as o(x,t) = 0.) When there is no t-dependence, this (n — m)-dimensional 
manifold S C R” is determined as the intersection of m (n — 1)-dimensional surfaces o;(x, t) = 0. These 
surfaces are designed in such a way that the system state trajectory, restricted to o(x, t) = 0, has a desired 
behavior such as stability or tracking. Although general nonlinear time-varying surfaces as in Equation 
50.3 are possible, linear ones are more prevalent in design [2-6]. Linear surface design is presented in 
Section 50.4. 


50.2.2 Control Structure 


After proper design of the surface, a controller u(x, t) = [ui (x, f),...,Um(x, t)]" is constructed, which 
generally has a switched form 


ae . 
Gish = : (x,t), when o;(x, t) > 0, (50.5) 


u; (x,t), when o;(x, t) <0. 


Equation 50.5 indicates that the control changes its value depending on the sign of o(x,t). Here 
we can define that a discontinuity set, D, in the right-hand side is a union of the hypersurfaces 
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defined by o;(x, t) = 0: 


D= les € R"|o;(x, t) = 0}. 


i=1 


Thus, the (possibly t-dependent) hypersurfaces {x € R"|o;(x, t) = 0} can be called switching surfaces 
and the functions o;(x,t) switching functions. The goal of phase 2 is to stabilize the state to S. Off S, 
the control values u; are chosen so that the state trajectory converges to S in finite time, that is, the 
sliding mode exists on S, but the sliding mode may (or may not) also exist on some of the hypersurfaces 
{x € R”|o;(x, t) = 0} while the state is converging to S. 


50.2.3 Sliding Modes 


The control u(x, f) is designed in such a way that the system state trajectory is attracted to S and, once 
having intercepted S, remains there for all subsequent time; thus, the state trajectory can be viewed as 
sliding along S meaning that the system is in a sliding mode. A sliding mode exists if, in the vicinity of 
the switching surface, S, the tangent or velocity vectors of the state trajectory point toward the switching 
surface. If the state trajectory intersects the sliding surface, the value of the state trajectory or “repre- 
sentative point” remains within an e-neighborhood of S. If a sliding mode exists on S, then S, or more 
commonly o(x, t) = 0, is also termed a sliding surface. Note that interception of the surface o;(x, t) = 0 
does not guarantee sliding on the surface for all subsequent time as illustrated in Figure 50.2, although 
this is possible. 

An ideal sliding mode exists only when the state trajectory x(t) of the controlled plant satisfies 
o(x(t),t) =0 at every tf > t; for some t). This may require infinitely fast switching. In real systems, a 
switched controller has imperfections, such as delay, hysteresis, and so on, which limit switching to a 
finite frequency. The representative point then oscillates within a neighborhood of the switching surface. 
This oscillation, called chattering (discussed in a later section), is also illustrated in Figure 50.2. If the 
frequency of the switching is very high relative to the dynamic response of the system, the imperfections 
and the finite switching frequencies are often but not always negligible. The subsequent development 
focuses primarily on ideal sliding modes. 


(Xp; to) 


Sliding x=0 


FIGURE 50.2 A situation in which a sliding mode exists on the intersection of the two indicated surfaces for t > fy. 


Variable Structure, Sliding-Mode Controller Design 50-5 


— evicinity of 
boundary point 


Non-sliding a 
oO 


trajectory 


Sliding mode 
trajectory 


Boundary point of D 


FIGURE 50.3 Two-dimensional illustration of the domain of a sliding mode. 


50.2.4 Conditions for the Existence of a Sliding Mode 


The existence of a sliding mode [2,5,6] requires stability of the state trajectory to the switching surface 
o(x, t) =0, that is, after some finite time ft), the system representative point, x(t), must be in some 
suitable neighborhood, {x| |]o(x, t)|| < e}, of S for suitable ¢ > 0. A domain, D, of dimension n— m in 
the manifold, S, is a sliding-mode domain if, for each ¢>0, there is a 8 > 0, so that any motion starting 
within an n-dimensional 8-vicinity of D may leave the n-dimensional ¢-vicinity of D only through the 
n-dimensional ¢-vicinity of the boundary of D as illustrated in Figure 50.3. 

The region of attraction is the largest subset of the state space from which sliding is achievable. A sliding 
mode is globally reachable if the domain of attraction is the entire state space. The second method of 
Lyapunov provides the natural setting for a controller design leading to a sliding mode. In this effort one 
uses a generalized Lyapunov function, V(t, x, 0), that is positive definite with a negative time derivative 
in the region of attraction. 


Theorem 50.1: [5, p. 83]: 


For the (n — m)-dimensional domain D to be the domain of a sliding mode, it is sufficient that in some 
n-dimensional domain Q D D, a function V(t, x, 0) exists, continuously differentiable with respect to all of 
its arguments and satisfying the following conditions: 


1. V(t,x, 0) is positive definite with respect to o, that is, for arbitrary t and x, V(t,x,0) > 0, wheno £0 
and V(t, x,0) = 0; on the sphere ||o|| < p > 0, for all x € Q and any t, the relationships 


inf V(t,x,0)=hp, hp >0 and sup V(t,x,o)=Hp, Hp >0 
IIoll=e lioll=o 


hold, where hy and Hy depend only on p with hy £ 0 if p £0. 

2. The total time derivative of V(t, x, 0) on the trajectories of the system of Equation 50.1 has a negative 
supremum for all x € Q except for x on the switching surface where the control inputs are undefined 
and the derivative of V(t, x, 0) does not exist. 
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In summary, two phases underlie VSC design. The first phase is to construct a switching surface 
o(x,t) =0, so that the system restricted to the surface has a desired global behavior, such as stability, 
tracking, regulation, and so on. The second phase is to design a (switched) controller u(x, t), so that away 
from the surface o(x, t) = 0, the tangent vectors of the state trajectories point toward the surface, that 
is, there is stability to the switching surface. This second phase is achieved by defining an appropriate 
Lyapunov function V(t,x,o), differentiating this function so that the control u(x,t) becomes explicit, and 
adjusting controller gains so that the derivative is negative definite. The choice of V(t,x,0) determines the 
allowable controller structures. Conversely, a workable control structure has a set of possible Lyapunov 
functions to verify its viability. A later section discusses general control structures. 


50.2.5 An Illustrative Example 


To conclude this section, we present an illustrative example for a single pendulum system, 


0 1 
x = A(x)x + Bu(x) = sin(x1) B + H u(x), 
——— 0 x2 1 
x) 
with a standard feedback control structure, u(x) = k,(x)x; + ko(x)x2, having nonlinear feedback gains 
switched according to the rule 
Ee aj(x), if o(x)x; > 0, 
Ow | Bilx), if o(x)x; <0, 


which depend on the linear switching surface (o(x) =[s] s2]x) . Without loss of generality, assume sz > 0. 
For such single-input systems it is ordinarily convenient to choose a Lyapunov function of the form 
V(t,x, 0) = 0.507(x). To determine the gains necessary to drive the system state to the surface o(x) = 0, 
they may be chosen so that 


; do? d 
VGx.6)=05 * = atx) =a(x)[s 1k 
= o(x)x E (x00 ss =) + o(x)axp [51 + s2k2(x)] <0. 
1 
This is satisfied whenever 
ay (x) = ay <m ea =-l, 
x1 
sin(x)) 
Bi (x) = Bi > max | |=» 
x1 XxX] 


dz < —(s1/s2) and Bz > —(s;/sz). Thus, for properly chosen s; and s2, the controller gains are readily 
computed. 

This example proposed no methodology for choosing s; and s2, that is, for designing the switching 
surface. Section 50.4 presents this topic. Further, this example was only single input. For the multi-input 
case, ease of computation of the control gains depends on a properly chosen Lyapunov function. For most 
cases, a quadratic Lyapunov function is adequate. This topic is discussed in Section 50.5. 


50.3 Existence and Uniqueness of Solutions to VSC Systems 


VSC produces system dynamics with discontinuous right-hand sides owing to the switching action of 
the controller. Thus they fail to satisfy conventional existence and uniqueness results of differential 
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F- 


F°=aF*+(1-a) FO 
ae [0,1] 


Ft 


FIGURE 50.4 Illustration of the Filippov method of determining the desired velocity vector F° for the motion of 
the state trajectory on the sliding surface as per Equation 50.6. 


equation theory. Nevertheless, an important aspect of VSC is the presumption that the plant behaves 
uniquely when restricted to o(x, t) = 0. One of the earliest and conceptually straightforward approaches 
addressing existence and uniqueness is the method of Filippov [7]. The following briefly reviews this 
method in the two-dimensional, single-input case. 

From Equation 50.1, x(t) = F(x,t,u) and the control u(x,t) satisfy Equation 50.5. Filippov’s work 
shows that the state trajectories of Equation 50.1 with control Equation 50.5 on the switching manifold 
Equation 50.3 solve the equation 


X(t) = aF*+ +(1—a)F- =F, 0<a<1. (50.6) 


This is illustrated in Figure 50.4, where Ft = F(x, t, ut), F~ = F(x, t,u_), and F° is the resulting velocity 
vector of the state trajectory in a sliding mode. 

The problem is to determine a, which follows from solving the equation (grad(c), F°) = 0, where the 
notation (a, b) denotes the inner product of a and J, that is, 


(grad(o), F~) 
o* (grad(o), (F- — F*))’ 


provided: 
1. (grad(o),(F~ — Ft)) £0. 
2. (grad(o), FT) <0. 
3. (grad(o), F~) > 0. 
Here, F° represents the “average” velocity, x(t) of the state trajectory restricted to o(x,t) =0. On 


average, the solution to Equation 50.1 with control Equation 50.5 exists and is uniquely defined on the 
switching surface S. This technique can also be used to determine the plant behavior in a sliding mode. 


50.4 Switching-Surface Design 


Switching-surface design is predicated based on the knowledge of the system behavior in a sliding mode. 
This behavior depends on the parameters of the switching surface. Nonlinear switching surfaces are 
nontrivial to design. In the linear case, the switching-surface design problem can be converted into an 
equivalent state feedback design problem. In any case, achieving a switching-surface design requires 
analytically specifying the motion of the state trajectory in a sliding mode. The so-called method of 
equivalent control is essential to this specification. 
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50.4.1 Equivalent Control 


Equivalent control constitutes a control input which, when exciting the system, produces the motion of 
the system on the sliding surface whenever the initial state is on the surface. Suppose at t, the plant’s 
state trajectory intercepts the switching surface and a sliding mode exists. The existence of a sliding 
mode implies that for all t > t),o(x(t), t) = 0, and hence 6(x(t), t) = 0. Using the chain rule, we define the 
equivalent control ue, for systems of the form of Equation 50.1 as the input, satisfying 


Oo 00, Oo 
x 


Oo Oo 
ot it Ox” Ot a axl & at ee eit: 


Assuming that the matrix product (00/0x)B(x, t) is nonsingular for all t and x, one can compute teq as 


Oo a0 80 
eq = —| =~ B(x, t — + f(x, Ff) }. 50.7 
Se E “ | (So + Fre ) Go) 
Therefore, given that o(x(t,), t;) = 0, then, for all t > t;, the dynamics of the system on the switching 
surface will satisfy 


-1 


-1 
x(t) = [ ~ B(x, | Sats, | | f(x, t) — Bex, | St, 5) ~. (50.8) 


This equation represents the equivalent system dynamics on the sliding surface. The driving term is present 
when some form of tracking or regulation is required of the controlled system, for example, when 


o(x, t) = Sx+r(t) =0 


with r(t) serving as a “reference” signal [4]. 

The (n — m)-dimensional switching surface, o(x, t) = 0, imposes m constraints on the plant dynamics 
in a sliding mode. Hence, m of the state variables can be eliminated, resulting in an equivalent reduced- 
order system whose dynamics represent the motion of the state trajectory in a sliding mode. Unfortunately, 
the structure of Equation 50.8 does not allow convenient exploiting of this fact in switching-surface 
design. To set forth a clearer switching-surface design algorithm, we first convert the plant dynamics to 
the so-called regular form. 


50.4.2 Regular Form of the Plant Dynamics 
The regular form of the dynamics of Equation 50.1 is 
= filet), 
: A : (50.9) 
22 = fa(z, t) + Ba(z, t)u(z, t), 


where z; € R"”™, z2 € R™. This form can often be constructed through a linear state transformation, 
z(t) = Tx(t), where T has the property 


TB(x, t) = TB(T~'z, t) = E . at 
2\4> 


and By(z, t) is an (m x m) nonsingular mapping for all f and z. In general, computing the regular form 
requires the nonlinear transformation, 


= Ton =|), 


T2(x, t) 
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where 


1. T(x,t) is a diffeomorphic transformation, that is, a continuous differentiable inverse mapping 
T(z, t) = x exists, satisfying T(0, t) = 0 for all t. 
. Ty6,:):R" x R-> R”™"™ and To(-,-): R” x R> R™. 
3. T(x,t) has the property that 


oT 
OT “Ox. a 0 
— B(x, t) = B(T(z,t),t) =| » 
ae lope ee ne 
Ox 


This partial differential equation has a solution only if the so-called Frobenius condition is satisfied 
[8]. The resulting nonlinear regular form of the plant dynamics has the structure, 


OT). youn 
a= Dye, t),t)+ =a * (28), 
= Se, £),¢) 4 eg ait, t), t) (50.10) 
A h(z,t ) + Bo(z, t)u. 


Sometimes all nonlinearities in the plant model can be moved to fl, t) so that 


a=filzt)= [An Ai] ak (50.11) 


which solves the sliding-surface design problem with linear techniques (to be shown). If the original 
system model is linear, the regular form is given by 


zy Ai Aj] | 2 0 
. |= + Uu, 50.12 
where z, € R"~™ and zz € R™ are as above. 


50.4.3 Equivalent System Dynamics via Regular Form 


The regular form of the equivalent state dynamics is convenient for analysis and switching-surface design. 
To simplify the development we make three assumptions: (1) the sliding surface is given in terms of the 
states of the regular form; (2) the surface has the linear time-varying structure, 


o(z,t)=Sz+r(t) =[S, S2] | +r(t)=0, 


where the matrix S2 is chosen to be nonsingular; and (3) the system is in a sliding mode, that is, for some 
t1, o(x(t), t) = 0 for all t > t;. With these three assumptions, one can solve for z2(t) as 


z(t) = —Sz ‘S21 (t) — Sy 'r(t). (50.13) 


Substituting Equation 50.13 into the nonlinear regular form of Equation 50.10 yields 


2 =filai,22,.t) =f (a, Sy'Siz1 s'r(),t) : 


The goal then is to choose S; and Sz to achieve a desired behavior of this nonlinear system. 
If this system is linear, that is, if Equation 50.11 is satisfied, then, using Equation 50.13, the reduced- 
order dynamics are 


A= (An —Ai28;151) Zl — A12S5'r(t). (50.14) 
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50.4.4 Analysis of the State Feedback Structure of Reduced-Order Linear 
Dynamics 


The equivalent reduced-order dynamics of Equation 50.14 exhibit a state feedback structure in which 
F=S, 'S is a state feedback map and Aj) represents an “input” matrix. Under the conditions that the 
original (linear) system is controllable, the following well-known theorem applies. 


Theorem 50.2: [9]: 


If the linear regular form of the state model (Equation 50.12) is controllable, then the pair (Ai, A12) of the 
reduced-order equivalent system of Equation 50.14 is controllable. 


This theorem leads to a wealth of switching-surface design mechanisms. First, it permits setting the 
poles of Ay; - A12S5 'S), for stabilizing the state trajectory to zero when r(f) = 0 or to a prescribed rate 
of tracking, otherwise. Alternatively, one can determine S; and S2 to solve the LQR (linear quadratic 
regulator) problem when r(t) = 0. 

As an example, suppose a system has the regular form of Equation 50.12 except that Az; and Az are 
time-varying and nonlinear, 


where aj = ajj(t,x) and an < aj(t,x) < ae, Let the switching surface be given by 


o(z) = [Sy S2] | = 0. 


Z] 
22 
The pertinent matrices of the reduced-order equivalent system matrices are 
0 1 0 0 0 
Ay = 0 0 0 and An= 1 O}. 
0 0 0 0 1 


To stabilize the system, suppose that the goal is to find F so that the equivalent system has eigenvalues 
at {-1,-2,-3}. Using MATLAB®’s Control System’s Toolbox yields 


el (eae ee 
ra[2 2 Jess, 


Choosing Sz = I leaves S; = F. This then specifies the switching-surface matrixS =[F I]. 
Alternatively, suppose that the objective is to find the control that minimizes the performance index 


[e,2) 
J= | (zi Qz + x Rix) dt, 
0 
where the lower limit of integration refers to the initiation of sliding. This is associated with the equivalent 


reduced-order system 
Z, = Az, — Aja, 
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where 
u = Sy Siz S Fz}. 


Suppose weighting matrices are taken as 


10 05 1.0 
2 0 
Q=|05 20 1.0], r=|7 a; 
10 1.0 3.0 


Using MATLAB Control Systems Toolbox, the optimal feedback is 


a 0.6420 1.4780 0.2230 
~ 10.4190 0.4461 1.7031] * 


Again, choosing Sy = I, the switching-surface matrix is given byS =[F I]. Here, the poles of the system 
in sliding are {-1.7742,-0.7034 +] 0.2623}. 


50.5 Controller Design 
50.5.1 Stability to Equilibrium Manifold 


As mentioned, in VSC a Lyapunov approach is used for deriving conditions on the control u(x, t) that 
will drive the state trajectory to the equilibrium manifold. Ordinarily, it is sufficient to consider only 
quadratic Lyapunov function candidates of the form 


V(t,x,0) = 0! (x, t)Wo(x, t), (50.15) 


where W is a symmetric positive-definite matrix. The control u(x,t) must be chosen so that the time 
derivative of V(t, x, 0) is negative definite for o 4 0. To this end, consider 


V(t,x,0) =6! Wot+o! W6 =20! Wo, (50.16) 


where we have suppressed specific x and t dependencies. Recalling Equation 50.1, it follows that 


. Oo Oo. Oo Oo ree (50.17) 
Og tan Bp oe oe 


Substituting Equation 50.17 into Equation 50.16 leads to a Lyapunov-like theorem. 


Theorem 50.3: 


A sufficient condition for the equilibrium manifold (Equation 50.3) to be globally attractive is that the 
control u(x, t) be chosen so that 


. Oo Oo Oo 
V =20'W— +207 W—f +207 W—Bu <0 50.18 
aioe ax! eax ( 
foro £0, that is, V(t, x, 0) is negative definite. 
Observe that Equation 50.18 is linear in the control. Virtually all control structures for VSC are chosen 


so that this inequality is satisfied for appropriate W. Some control laws utilize an x- and t-dependent W 
requiring that the derivation above be generalized. 
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50.5.2 Various Control Structures 


To make the needed control structures more transparent, recall the equivalent control of Equation 50.7, 


Oo 


Oo =I Oo 
Ueg(x, t) = -| Fa ) (= + Bal & ») 


computed assuming that the matrix product 2 B(x,t) is nonsingular for all t and x. We can now decom- 
pose the general control structure as 


u(x, t) = Ue (X; t) + un (x, t), (50.19) 
where un (x, t) is as yet an unspecified substructure. Substituting the above into Equation 50.18 produces 
the following sufficient condition for stability to the switching surface: Choose un (x, t) so that 


a 
Vikee)= 20" W Bl, t)uy (x,t) <0. (50.20) 
Xx 


Because & B(x, t) is assumed to be nonsingular for all ¢ and x, it is convenient to set 


-1 
un (x, t) = | Seat a| un (x, t). (50.21) 


Often a switching surface o(x,t) can be designed to achieve a desired system behavior in sliding and, 
at the same time, to satisfy the constraint 2B = I in which case uy = tin. Without loss of generality, we 
make one last simplifying assumption, W = I, because W > 0, W is nonsingular, and can be compensated 
for in the control structure. Hence, stability on the surface reduces to finding tin (x, t) such that 


, T Oo do Jol. Ta 

V=20°W|—B||—B] un=20' un <0. (50.22) 
Ox Ox 

These simplifications allow us to specify five common controller structures: 


1. Relays with constant gains: iy (x, t) is chosen so that 
un = a sgn(o(x, t)) 


with a = [aj] an m x m matrix, and sgn (o(x, t)) is defined componentwise. Stability to the sur- 
face is achieved if a =[ajj] is chosen diagonally dominant with negative diagonal entries [5]. 
Alternatively, if a is chosen to be diagonal with negative diagonal entries, then the control can be 
represented as 

iin; = ajjsgn(o;(x, t)) 


and, for 0; £0, 
2ojtni = 2ajo;sgn(o;) = 2a;j|o;| < 0, 


which guarantees stability to the surface, given the Lyapunov function, V(t, x, 0) = 01 (x, t)o(x, f). 
2. Relays with state-dependent gains: Each entry of ity (x, t) is chosen so that 


Uni = a(x, t)sgn(oi(x,t)), a(x, t) <0. 
The condition for stability to the surface is that 
2ojtin = 204i(x, thoisgn(oj) = 2a4(x, t)|o;] <0 foro; 40. 
An adequate choice for «j;(x, t) is to choose Bj < 0, yj > 0, and k a natural number with 


atii(x) = Bi(o?* (x, t) + yi). 
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3. Linear state feedback with switched gains: Here iy (x,t) is chosen so that 


Qij <0, OjX; > 0, 


un= Wx; V=(V;); Y= 
il : Bi > 0, oj; <0. 


To guarantee stability, it is sufficient to choose aj and Bj so that 
oj itn; = 0; (Wier + Winx, +--+ + Winxn) = Voix, + Vj20;x2 +--+ + Vinoixn < 0. 


4. Linear continuous feedback: Choose 


A 


iy = —Po(x,t), P=P!>0, 


that is, P € R™*"™ is a symmetric positive-definite constant matrix. Stability is achieved because 


o in =—o! Po <0, 


where P is often chosen as a diagonal matrix with positive diagonal entries. 
5. Univector nonlinearity with scale factor: In this case, choose 


o(x, t) 
ade Vinee 
tin = ¢ |lo(x, || 
0, o=0. 


p<0O and o#0, 


Stability to the surface is guaranteed because, for o # 0, 


Ta oo 
o° un = ——p=|lol|p < 0. 
Ilol| 


Of course, it is possible to make p time dependent, if necessary, for certain tracking problems. This 
concludes our discussion of control structures to achieve stability to the sliding surface. 


50.6 Design Examples 


This section presents two design examples illustrating typical VSC strategies. 


Design Example 50.1: 


In this example, we illustrate a constant gain relay control with nonlinear sliding surface design for a 
single-link robotic manipulator driven by a dc armature-controlled motor modeled by the normalized 
(i.e, scaled) simplified equations 


xy X2 0 
X2 | =| sin(x;)+x3 } +] 0] u=f(x)+Bu 
x3 X2 +X3 1 


in the regular form. 
To determine the structure of an appropriate sliding surface, recall the assumption that 98 Bix, t) 


is nonsingular. Because B = [0 0 1]/, it follows that ie. must be nonzero. Without losing generality, 
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we set fe = 1.Hence, it is sufficient to consider sliding surfaces of the form 
o(X) = 0(X1,X2,X3) = 01 (x1,X2) + x3 = 0. (50.23) 


Our design presumes that the reduced-order dynamics have a second-order response represented 
by the reduced-order state dynamics, 


xy} x2 _ | 0 1 x1 
x} L-a1x1 — 2x2] [-a1 —a2] [x2 |" 


This form allows us to specify the characteristic polynomial of the dynamics and thus the eigenvalues, 
that is, w4(A) = 42 + a2 +}. Proper choice of a; and ap leads to proper rise time, settling time, 
overshoot, gain margin, and so on. 

The switching-surface structure of Equation 50.23 implies that, in a sliding mode, 


X3 = —04(X1,X2). 


Substituting the above equation into the given system model, the reduced-order dynamics become 


> [soil oem Sam Cael 
x2] | sin(xy) — 01 (x1,x2)} | -a1x1 — aax2 |" 


Hence the switching-surface design is completed by setting 
04 (X1,X2) = sin(xy) + 41X17 + 42X2. 


To complete the controller design, we first compute the equivalent control, 


Oo, 004 1 ; x2 
[EB fate 
2 3 


For the constant gain relay control structure (Equation 50.19), 
UN = aSgN(o(x)). 
Stability to the switching surface results whenever a <0 as 


06 = ao sgn(o) = alo| < 0. 


Design Example 50.2: 


Consider the fourth-order (linear) model of a mass-spring system that could represent a simplified 
model of a flexible structure in space with two-dimensional control (Figure 50.5). 


aa 


al x2 


uy 


FIGURE 50.5 A mass spring system for design Example 50.2. 
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Here, x1 is the position of m1, x2 the position of m2, u; the force applied to m1, and u2 the force 
applied between m, and mp. The differential equation model has the form 


m4X4 +k(xy — X2) =U, +U, 
mx2 + k(x2 — x1) = —u2, 


where k is the spring constant. Given that x3 = x1 and x4 = x2, the resulting state model in regular 
form is 


|_| k | 
- ——x1 + —x2 | +} —u, + —u2 
x3 m m | my m | 
X4 k 1 . | 
—x; —- —x. ——u2 
m2 m2 m2 


There are two simultaneous control objectives: 


1. Stabilize oscillations, that is, x; = x2, 
2. Track a desired trajectory, x2(t) = Zye¢(t). 


These goals are achieved if the following relationships are maintained for c1, cz > 0: 
xy —X2 +00] —X2) = 0 =} X1-x2— 0 
and 
X2 — Zref + €2(X2 — Zref) =O => x2 —Zref —> 0. 


The first step is to determine the appropriate sliding surface. To achieve the first control objective, 
set 


04 (x, t) = x3 — X4 +€1 (x1 — x2) = 0, 


and to achieve the desired tracking, set 
62(x, t) = x4 — Zpe¢ + €2(X2 — Zyef) = 0. 

The next step is to design a VSC law to drive the state trajectory to the intersection of these switching 
surfaces. In this effort, we illustrate two controller designs. The first is a hierarchical structure [2] so 
that, foro 40, 

uy= a71sgn(o7), 

U2 = a2SgN(o2) 
with the sign of «1, a2 4 0 to be determined. 

For stability to the surface, it is sufficient to have 0,6, < 0 and 0262 < 0, as can be seen from 
Equation 50.16, with W = /. Observe that 
61 = X3 — X4 + C4 (0% — Xz) = X3 — X4 + €4 (x3 — XQ) 
and 
52 = X4 — Zrep + €2(X2 — Zref) = X4 — Zrep + €2(X4 — Zref). 

Substituting for the derivatives of x3 and x4 leads to 


1 1 1 1 1 1 


k |- my om mM) [in| Heal _{|m mm Bea ak (50 24) 
62 ‘i els u2 ho 0 va a2Sgn(o2) ho 
m2 m2 
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where 
k k k k 
h= Xi + X2 X1 + X2 + €1X3 — €1X4 
m mM m2 m2 
and 
h k ee : 
= —X] — —x2--Z, C2X4 — C2Zyef- 
2 m) 1 m> 2 ref 2X4 2¢ref 


Taking a brute force approach to the computation of the control gains, stability to the switching 
surface is achieved, provided 


—a 
0262 = —* gy sgn(o>) +o 2h? <0, 
m2 
which is satisfied if 


a2 > m2|h2|(> 0), 
and provided 


a 1 1 
010, = ml sgn(o1) +04 (= + —~) aasanton) +h <0 


which is satisfied if 


m 
ay << —m4{|hq| T+ Q2. 
m2 
In a second controller design, we recall Equation 50.24. For 0 4 0 and o27 £0, it is convenient to 
define the controller as 


1 1 17471 


len |= m mm) eel 
u2 ) a Basgn(o2) |’ 


m2 


where B, and B2 are to be determined. It follows that 


61] _ |Bisgnlor) |, | 
52 Basgn(o2)| | h2 |” 
As in the first controller design, the state trajectory will intercept the sliding surface in finite time and 


sliding will occur for 8; and B2 sufficiently negative, thereby achieving the desired control objective. 


A unifying characterization of sliding mode controllers that drive a trajectory to the sliding manifold 
is given in [10]. 


50.7 Chattering 


The VSC controllers developed earlier assure the desired behavior of the closed-loop system. These 
controllers, however, require an infinitely (in the ideal case) fast switching mechanism. The phenomenon 
of nonideal but fast switching was labeled as chattering (actually, the word stems from the noise generated 
by the switching element). The high-frequency components of the chattering are undesirable because they 
may excite unmodeled high-frequency plant dynamics resulting in unforeseen instabilities. To reduce 
chatter, define a so-called boundary layer as 


{x| |lo(x)|] < ¢,¢ > 0}, (50.25) 


whose thickness is 28. Now, modify the control law of Equation 50.26 (suppressing t and x arguments) to 


Ueg + UN, o|| = &, 
c= eq ‘N l| | = (50.26) 
eq + p(o,x), |loll < & 
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where p(o,x) is any continuous function satisfying p(0,x) = 0 and p(o, x) = uy(x) when |lo(x)|| =. 
This control guarantees that trajectories are attracted to the boundary layer. Inside the boundary layer, 
Equation 50.26 offers a continuous approximation to the usual discontinuous control action. Similar to 
Corless and Leitmann [11], asymptotic stability is not guaranteed but ultimate boundedness of trajectories 
to within an e-dependent neighborhood of the origin is assured. 


50.8 Robustness to Matched Disturbances and Parameter 
Variations 


To explore the robustness of VSC to disturbances and parameter variations, one modifies Equation 50.1 
to 


x(t) = [f(x, t) + Af (x, t, q(t))] + [B(x, t) + AB(x, t, q(t))]u(x, t) + d(t), (50.27) 


where q(t) is a vector function representing parameter uncertainties, Af and AB represent the cumulative 
effects of all plant uncertainties, and d(t) denotes an external (deterministic) disturbance. The first critical 
assumption in our development is that all uncertainties and external disturbances satisfy the so-called 
matching condition, that is, Af, AB, and d(t) lie in the image of B(x, t) for all x and t. As such they can all 
be lumped into a single vector function E(x, t, q, d, u), so that 


x(t) = f (x, t) + B(x, thu(x, t) + B(x, t)E(x, t, q, d, u). (50.28) 

The second critical assumption is that a positive continuous bounded function p(x, f) exists, satisfying 
I|5(x,t,q,d,u)|| < p(x, t). (50.29) 
To incorporate robustness into a VSC design, we utilize the control structure of Equation 50.19, u(x, t) = 


Ueq (x, t) + un (x, t), where teq(x, t) is given by Equation 50.7. Given the plant and disturbance model of 
Equation 50.28, then, as per Equation 50.20, it is necessary to choose un (x, t), so that 


V(t,x,0) = 207 WOO, t) [un (x, t) +&(x, t, q, d, u)| <0. 
x 


Choosing any one of the control structures outlined in Section 50.5, a choice of sufficiently “high” gains 
will produce a negative-definite V(t, x, 0). Alternatively, one can use a control structure [2], 


Br 9275 
— 2x! fox, t)+a(x,t)] foro(x,t) 40, 
uv(s= 4 [Bt[22]"o| (50.30) 
0 otherwise, 


where a(x, f) is to be determined. Assuming W = I, it follows that, for o 4 0, 


x [o(x, t) + a(x, 1] 4 rot 2 Be 
Ox 


Choosing a(x, t) = a > 0 leads to the stability of the state trajectory to the equilibrium manifold despite 
matched disturbances and parameter variations, demonstrating the robustness property of a VSC law. 
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50.9 Observer Design 


“Observers” can be viewed as software algorithms that allow online estimation of the current state of a 
dynamic system when only the output and the input of the system can be measured. In the case of a linear 
system, we have 

x= Ax+ Bu+ Bé, 

(50.31) 

y=Cx, 
where C € R?*", and we assume that the pair (C, A) is observable. The observer design problem is to 
construct a dynamic system that estimates the system state based on knowledge of the input and the 
output measurement. This results in the so-called Luenberger observer when & = 0, 


& = AX + Bu+ L(y — CR), (50.32) 


The estimation error e(t) = x(t) — X(t) satisfies: e(t) = (A — LC)e(t). Since (C, A) is observable, the eigen- 
values of A — LC can be assigned arbitrarily by a choice of the gain matrix L, although in practice this is 
limited by the bandwidth of the system. 

The sliding mode concept can be used for designing an observer by replacing L(y — Cx) in Equation 
50.32 with a discontinuous function Eq(y, x) of and x yielding 


= ae EEE ay (50.33) 
y=Cx, 
where Ey is a user-chosen function to insure convergence in the presence of uncertainties modeled by 
nonzero § in Equation 50.31. 
One possibility is to choose Eg(y, ¥) = L(y — Cx) + BE(y, 9), where L is chosen so that A-LC isa stability 
matrix (eigenvalues in the open left-half-complex plane) and 


; Fy —y) 
Ey.) =n ————— 
Yy=n [Fy 5) 
where n is a design parameter satisfying y > ||&||. Now, L, F ¢ R”*?, p> m, and a matrix P= P’ > 0 
must simultaneously satisfy: 
1. eig(A — LC) C C™ 


2. FC=B'P, and 
3. (A—LC)'P+P(A—LC) =—Q 


, (50.34) 


for an appropriate Q = Q? >0, if it exists. A solution for (L, F, P) exists if and only if 
1. rank(B) = rank(CB) = r and 


2. rank fae a =n-+r,Re[s] > 0 


With the estimation error as e(t) = x(t) — X(t), the error dynamics become 
e(t) = (A — LC)e(t) — BE(y, y) + BE. (50.35) 
It follows that 
£ (e" Pe) = —e! Qe — 21 ||FCe|| + 2e7 PBE 
< —e" Qe — 2n || FCel| + 2 || FCe|| ||§|| < —e” Qe, 


which implies lim;_, .e(t) = 0. For further analysis see [3,12,13]. For an alternate sliding mode observer 
structure, see [14-16]. 
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50.9.1 Observer Design 2 [17,18] 
Now consider Eq(y, x) = L sign(y — Cx) resulting in the observer dynamics 
x = Ax + Bu+ Lsign(y — Cx). (50.36) 


For the deterministic case (€ = 0) the observation error satisfies e = Ae — L sgn(Ce). For such a system, 
a sliding mode is possible on the manifold Ce = 0. In order to describe the choice of the observer gain L 
and analyze the error dynamics let us consider a nonsingular transformation of the state x into a new set 
of coordinates such that the first p coordinates correspond to the observed vector y: 


The transformed plant dynamics are 


vy) |An Ally By 
H be — H as A ue (50.37) 


The observer in the new coordinates is 


y=Any Aypwt Buut+L, sen(y — y), (50.38a) 


W = Any + Any + Bou + Ly sgn(y — J). (50.38b) 

Denoting e(t) = y(t) — p(t) and e2(t) = w(t) — w(t), the error dynamics for the first subsystem is 
€) = Aye; + Ajzen — Li sgn(e), (50.39a) 
€2 = Ape) + Anze2 — Lo sgn(ey). (50.39b) 


By choosing an appropriate nonsingular gain matrix L) (it is a square matrix), we can enforce sliding 
regime in the first equation along the manifold e;(t) = 0. Indeed, the equivalent control is obtained from 
Equation 50.39a under the condition, é)(t) = 0 as 


[sgn(e1)]., = Ly Ape. (50.40) 


The dynamics of the system in a sliding mode (e, = 0) can be obtained by substituting this value into 
Equation 50.39b, to obtain the linear equation 


é2 = (Aza — L217 'Ai2)e2. (50.41) 


Let us note that the observability of the original pair (C, A) implies observability of the pair (Aj2, A22) 
in the system (Equation 50.37). Using this fact it follows that we can assign any eigenvalues in this system 
by appropriate choice of Lz; thus, guaranteeing convergence e2(t) > 0 with any desired exponential 
rate. The dimension of the system or Equation 50.41 is n—p. The case when the output is corrupted by 
measurement noise was also considered in [18]. Similar observer structures and explanations can be 
found in [6,14-16]. An application of such an observer structure to state estimation of a magnetic bearing 
is considered in [19]. 

In [20], Drakunov proposed a sliding mode observer structure 


. [OH()]"! 
i= | | M(&, t) sgn[V — H(2)] (50.42) 
x 
that can be used for a nonlinear system of the form 
x= f(x), 
y=h(x), 
where the measurement map h:R"— R is a scalar and where H(x) = [hy(x) ho(x) --- hy(x)]" 


has entries defined using repeated Lie derivatives: h(x) = h(x), ha(x) = Lrh(x), h3(x) = 
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L¢h(x), wo Ay(x) = Lf *h(x)s M(x, t) = diag(m,(%, t),...,mn(x,t)) is a diagonal gain matrix and 
the vector V=[v, v2 --- val? has components defined recursively: v)(t)= y(t), vi4i(t) = 
[mj(%, t)sgn (vi(t) _ hi(&)) 4° The equivalent values can be obtained using an equivalent control fil- 
ter such as a low-pass filter, although a first-order low-pass filter may not be sufficient; more complicated 
even nonlinear digital filters may need to be employed. 


Example 50.3: 


To illustrate the above nonlinear observer design, consider the nonlinear state model 


iq = (1 — 2xy +. 2x3)x2, 


xX) = —Xq +x3 


with the output y = x, — x4. In this case, we have h(x) = hy(x1,X2) = x1 — x3, and since n=2 we 
need only the first Lie derivative: hz(x1,x2) = Lgh(x) = x2. Therefore, the corresponding map H and 


its Jacobian matrix are 
oH = _ {1 2x2 
Ox “10 14° 


_ x2 OH = 
Ho " 2], On lo fl 
% = my sgnly — 1 + X3) + 2m2% sgn(v — %), 


The observer of Equation 50.42 is 


x =m)? sgn(v — xX), 


where v = {m; sgn(y — x1 + 8 )heq. The second-order observer converges as long as the observer 
gains are sufficiently large, which means that m, > |x2|,m2 > Ik —x3|, If the region of initial condi- 


tions and system trajectories are bounded, then the gains can be chosen to be constant. In general, 
the gains depend on (x7, X). The equivalent value operator {-- - }eq can be implemented in different 


FIGURE 50.6 Nonlinear observer convergence. 
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ways, the easiest of which is a first-order low-pass filter v = —Av + 4m 1sgn(y — xX + x2) for an appro- 
priate value of \ > 0. The resulting solution to the filtering equation is v = {m;sgnly — xy + )heq. 
The simulation results are shown in Figure 50.6. 


For further work on VSC systems and sliding mode observers we refer the reader to [16,20-22]. 


50.10 Concluding Remarks 


This chapter has summarized the salient results of sliding mode control theory and illustrated the design 
procedures with various examples. A wealth of literature exists on the subject that cannot be included 
because of space limitations. In particular, the literature is replete with realistic applications [1,23], 
extensions to output feedback [24], extensions to decentralized control [25], and extensions to discrete- 
time systems. Additionally, there is some work, old and new, on higher-order sliding modes [16] and 
[26]. For extensions of the above methods to time delay systems, see [18,27,29]. The reader is encouraged 
to search the literature for many papers in this area. 


50.11 Defining Terms 


Chattering: The phenomenon of nonideal but fast switching. The term stems from the noise generated 
by a switching element. 

Equilibrium (discontinuity) manifold: A specified, user-chosen manifold in the state space to which a 
system’s trajectory is driven and maintained for all time subsequent to intersection of the mani- 
fold by a discontinuous control that is a function of the system’s states, and hence, discontinuity 
manifold. Other terms commonly used are sliding surface and switching surface. 

Equivalent control: The solution to the algebraic equation involving the derivative of the equation of the 
switching surface and the plant’s dynamic model. The equivalent control is used to determine 
the system’s dynamics on the sliding surface. 

Equivalent system dynamics: The system dynamics obtained after substituting the equivalent control 
into the plant’s dynamic model. It characterizes state motion parallel to the sliding surface if the 
system’s initial state is off the surface and state motion is on the sliding surface if the initial state 
is on the surface. 

Ideal sliding mode: Motion of a system’s state trajectory along a switching surface when switching in the 
control law is infinitely fast. 

Matching condition: The condition requiring the plant’s uncertainties to lie in the image of the input 
matrix, that is, the uncertainties can affect the plant dynamics only through the same channels 
as the plant’s input. 

Region of attraction: A set of initial states in the state space from which sliding is achievable. 

Regular form: A particular form of the state-space description of a dynamic system obtained by a suitable 
transformation of the system’s state variables. 

Sliding surface: See equilibrium manifold. 

Switching surface: See equilibrium manifold. 
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51.1 Introduction 


This chapter deals with the control of bifurcations and chaos in nonlinear dynamical systems. This is 
a young subject area that is currently in a state of active development. Investigations of control system 
issues related to bifurcations and chaos began relatively recently, with most currently available results 
having been published within the past decade. Given this state of affairs, a unifying and comprehensive 
picture of control of bifurcations and chaos does not yet exist. Therefore, the chapter has a modest but, it 
is hoped, useful goal: to summarize some of the motivation, techniques, and results achieved to date on 
control of bifurcations and chaos. Background material on nonlinear dynamical behavior is also given, 
to make the chapter somewhat self-contained. However, interested readers unfamiliar with nonlinear 
dynamics will find it helpful to consult nonlinear dynamics texts to reinforce their understanding. 
Despite its youth, the literature on control of bifurcations and chaos contains a large variety of 
approaches as well as interesting applications. Only a small number of approaches and applications 
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can be touched upon here, and these reflect the background and interests of the authors. The Further 
Reading section provides references for those who would like to learn about alternative approaches or to 
learn more about the approaches discussed here. 

Control system design is an enabling technology for a variety of application problems in which non- 
linear dynamical behavior arises. The ability to manage this behavior can result in significant practical 
benefits. This might entail facilitating system operability in regimes where linear control methods break 
down; taking advantage of chaotic behavior to capture a desired oscillatory behavior without expending 
much control energy; or purposely introducing a chaotic signal in a communication system to mask a 
transmitted signal from an adversary while allowing perfect reception by the intended party. 

The control problems addressed in this chapter are characterized by two main features: 


1. Nonlinear dynamical phenomena impact system behavior 
2. Control objectives can be met by altering nonlinear phenomena 


Nonlinear dynamics concepts are clearly important in understanding the behavior of such systems (with 
or without control). Traditional linear control methods are, however, often effective in the design of control 
strategies for these systems. In other cases, such as for systems of the type discussed in Section 51.5.2, 
nonlinear methods are needed both for control design and performance assessment. 

The chapter proceeds as follows. In Section 51.2, some basic nonlinear system terminology is recalled. 
This section also includes a new term, namely “candidate operating condition,” which facilitates sub- 
sequent discussions on control goals and strategies. Section 51.3 contains a brief summary of basic 
bifurcation and chaos concepts that will be needed. Section 51.4 provides application examples for which 
bifurcations and/or chaotic behavior occur. Remarks on the control aspects of these applications are also 
given. Section 51.5 is largely a review of some basic concepts related to control of parameterized families 
of (nonlinear) systems. Section 51.5 also includes a discussion of what might be called “stressed operation” 
of a system. Section 51.6 is devoted to control problems for systems exhibiting bifurcation behavior. The 
subject of Section 51.7 is control of chaos. Conclusions are given in Section 51.8. The final section gives 
some suggestions for further reading. 


51.2 Operating Conditions of Nonlinear Systems 


In linear system analysis and control, a blanket assumption is made that the operating condition of interest 
is a particular equilibrium point, which is then taken as the origin in the state space. The topic addressed 
here relates to applying control to alter the dynamical behavior of a system possessing multiple possible 
operating conditions. The control might alter these conditions in terms of their location and amplitude, 
and/or stabilize certain possible operating conditions, permitting them to take the place of an undesirable 
open-loop behavior. For the purposes of this chapter, therefore, it is important to consider a variety of 
possible operating conditions, in addition to the single equilibrium point focused on in linear system 
theory. In this section, some basic terminology regarding operating conditions for nonlinear systems is 
established. 
Consider a finite-dimensional continuous-time system 


x(t) = F(x(t)) (51.1) 


where x € R” is the system state and F is smooth in x. (The terminology recalled next extends straightfor- 
wardly to discrete-time systems x*+! = F(x*).) An equilibrium point or fixed point of the system (Equation 
51.1) isa constant steady-state solution, i.e., a vector x° for which F(x°) = 0. A periodic solution of the sys- 
tem is a trajectory x(t) for which there isa minimum T > 0 such that x(t + T) = x(t) for all t. An invariant 
set is a set for which any trajectory starting from an initial condition within the set remains in the set for 
all future and past times. An isolated invariant set is a bounded invariant set a neighborhood of which 
contains no other invariant set. Equilibrium points and periodic solutions are examples of invariant sets. 
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A periodic solution is called a limit cycle if it is isolated. An attractor is a bounded invariant set to which 
trajectories starting from all sufficiently nearby initial conditions converge as t + oo. The positive limit 
set of Equation 51.1 is the ensemble of points that some system trajectory either approaches as t —> 00 or 
makes passes nearer and nearer to as t > oo. For example, ifa system is such that all trajectories converge 
either to an equilibrium point or a limit cycle, then the positive limit set would be the union of points on 
the limit cycle and the equilibrium point. The negative limit set is the positive limit set of the system run 
with time t replaced by —t. Thus, the positive limit set is the set where the system ends up at t = +00, 
while the negative limit set is the set where the system begins at t = —oo [30]. The limit set of the system 
is the union of its positive and negative limit sets. 

It is now possible to introduce a term that will facilitate the discussions on control of bifurcations and 
chaos. A candidate operating condition of a dynamical system is an equilibrium point, periodic solution 
or other invariant subset of its limit set. Thus, a candidate operating condition is any possible steady-state 
solution of the system, without regard to its stability properties. This term, though not standard, is useful 
since it permits discussion of bifurcations, bifurcated solutions, and chaotic motion in general terms 
without having to specify a particular nonlinear phenomenon. The idea behind this term is that such a 
solution, if stable or stabilizable using available controls, might qualify as an operating condition of the 
system. Whether or not it would be acceptable as an actual operating condition would depend on the 
system and on characteristics of the candidate operating condition. As an extreme example, a steady spin 
of an airplane is a candidate operating condition, but certainly is not acceptable as an actual operating 
condition! 


51.3 Bifurcations and Chaos 


This section summarizes background material on bifurcations and chaos that is needed in the remainder 
of the chapter. 


51.3.1 Bifurcations 


A bifurcation is a change in the number of candidate operating conditions of a nonlinear system that 
occurs as a parameter is quasistatically varied. The parameter being varied is referred to as the bifurcation 
parameter. A value of the bifurcation parameter at which a bifurcation occurs is called a critical value of 
the bifurcation parameter. Bifurcations from a nominal operating condition can only occur at parameter 
values for which the condition (say, an equilibrium point or limit cycle) either loses stability or ceases to 
exist. 

To fix ideas, consider a general one-parameter family of ordinary differential equation systems 


x = F(x) (51.2) 


where x € R” is the system state, jt € R denotes the bifurcation parameter, and F is smooth in x and w. 
Equation 51.2 can be viewed as resulting from a particular choice of control law in a family of nonlinear 
control systems (in particular, as Equation 51.9 with the control u set to a fixed feedback function u(x, )). 
For any value of 1, the equilibrium points of Equation 51.2 are given by the solutions for x of the algebraic 
equations F(x) = 0. 

Local bifurcations are those that occur in the vicinity of an equilibrium point. For example, a small- 
amplitude limit cycle can emerge (bifurcate) from an equilibrium as the bifurcation parameter is varied. 
The bifurcation is said to occur regardless of the stability or instability of the “bifurcated” limit cycle. 
In another local bifurcation, a pair of equilibrium points can emerge from a nominal equilibrium point. 
In either case, the bifurcated solutions are close to the original equilibrium point—hence the name local 
bifurcation. Global bifurcations are bifurcations that are not local, ie., those that involve a domain in 
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phase space. Thus, if a limit cycle loses stability releasing a new limit cycle, a global bifurcation is said to 
take place.* 

The nominal operating condition of Equation 51.2 can be an equilibrium point or a limit cycle. In fact, 
depending on the coordinates used, it is possible that a limit cycle in one mathematical model corresponds 
to an equilibrium point in another. This is the case, for example, when a truncated Fourier series is used 
to approximate a limit cycle, and the amplitudes of the harmonic terms are used as state variables in 
the approximate model. The original limit cycle is then represented as an equilibrium in the amplitude 
coordinates. 

If the nominal operating condition of Equation 51.2 is an equilibrium point, then bifurcations from 
this condition can occur only when the linearized system loses stability. Suppose, for example, that the 
origin is the nominal operating condition for some range of parameter values. That is, let F"(0) = 0 for 
all values of t for which the nominal equilibrium exists. Denote the Jacobian matrix of Equation 51.2 
evaluated at the origin by 


OF" 
A(L) = py: 


Local bifurcations from the origin can only occur at parameter values \t where A(\) loses stability. 
The scalar differential equation 
k= px—x? (51.3) 


provides a simple example of a bifurcation. The origin x° = 0 is an equilibrium point for all values of the 
real parameter jt. The Jacobian matrix is A(jt) = (a scalar). It is easy to see that the origin loses stability 
as |t increases through yw = 0. Indeed, a bifurcation from the origin takes place at 4. = 0. For p <0, 
the only equilibrium point of Equation 51.3 is the origin. For 1. > 0, however, there are two additional 
equilibrium points at x = +,/jt. This pair of equilibrium points is said to bifurcate from the origin at 
the critical parameter value \., = 0. This is an example of a pitchfork bifurcation, which will be discussed 
later. 


51.3.1.1 Subcritical vs. Supercritical Bifurcations 


In a very real sense, the fact that bifurcations occur when stability is lost is helpful from the perspective 
of control system design. To explain this, suppose that a system operating condition (the “nominal” 
operating condition) is not stabilizable beyond a critical parameter value. Suppose a bifurcation occurs 
at the critical parameter value. That is, suppose a new candidate operating condition emerges from the 
nominal one at the critical parameter value. Then it may be that the new operating condition is stable 
and occurs beyond the critical parameter value, providing an alternative operating condition near the 
nominal one. This is referred to as a supercritical bifurcation. In contrast, it may happen that the new 
operating condition is unstable and occurs prior to the critical parameter value. In this situation (called 
a subcritical bifurcation), the system state must leave the vicinity of the nominal operating condition for 
parameter values beyond the critical value. However, feedback offers the possibility of rendering such a 
bifurcation supercritical. This is true even if the nominal operating condition is not stabilizable. If such a 
feedback control can be found, then the system behavior beyond the stability boundary can remain close 
to its behavior at the nominal operating condition. 

The foregoing discussion of bifurcations and their implications for system behavior can be gainfully 
viewed using graphical sketches called bifurcation diagrams. These are depictions of the equilibrium 
points and limit cycles of a system plotted against the bifurcation parameter. A bifurcation diagram is a 
schematic representation in which only a measure of the amplitude (or norm) of an equilibrium point or 
limit cycle need be plotted. In the bifurcation diagrams given in this chapter, a solid line indicates a stable 
solution, while a dashed line indicates an unstable solution. 


* This use of the terms local bifurcation and global bifurcation is common. However, in some books, a bifurcation from a 


limit cycle is also referred to as a local bifurcation. 
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FIGURE 51.1 Subcritical bifurcation with hysteresis. 


Several bifurcation diagrams will now be used to further explain the meanings of supercritical and 
subcritical bifurcation, and to introduce some common bifurcations. It should be noted that not all 
bifurcations are supercritical or subcritical. For example, bifurcation can also be transcritical. In such 
a bifurcation, bifurcated operating conditions occur both prior to and beyond the critical parameter 
value. Identifying a bifurcation as supercritical, subcritical, transcritical, or otherwise is the problem of 
determining the direction of the bifurcation. A book on nonlinear dynamics should be consulted for a more 
extensive treatment (e.g., [10,18,28,30,32,43,46]). In this chapter only the basic elements of bifurcation 
analysis can be touched upon. 

Suppose that the origin of Equation 51.2 loses stability as u increases through the critical parameter 
value L = |t-. Under mild assumptions, it can be shown that a bifurcation occurs at [L¢. 

Figure 51.1 serves two purposes: it depicts a subcritical bifurcation from the origin, and it shows a 
common consequence of subcritical bifurcation, namely hysteresis. A subcritical bifurcation occurs from 
the origin at the point labeled A in the figure. It leads to the unstable candidate operating condition 
corresponding to points on the dashed curve connecting points A and B. As the parameter 1 is decreased 
to its value at point B, the bifurcated solution merges with another (stable) candidate operating condi- 
tion and disappears. A saddle-node bifurcation is said to occur at point B. This is because the unstable 
candidate operating condition (the “saddle” lying on the dashed curve) merges with a stable candidate 
operating condition (the “node” lying on the solid curve). These candidate operating conditions can be 
equilibrium points or limit cycles—both situations are captured in the figure. Indeed, the origin can also 
be reinterpreted as a limit cycle and the diagram would still be meaningful. Another common name for a 
saddle-node bifurcation point is a turning point. 

The physical scenario implied by Figure 51.1 can be summarized as follows. Starting from operation at 
the origin for small |, increasing | until point A is reached does not alter system behavior. If is increased 
past point A, however, the origin loses stability. The system then transitions (“jumps”) to the available 
stable operating condition on the upper solid curve. This large transition can be intolerable in many 
applications. As jz is then decreased, another transition back to the origin occurs but at a lower parameter 
value, namely that corresponding to point B. Thus, the combination of the subcritical bifurcation at A 
and the saddle-node bifurcation at B can lead to a hysteresis effect. 

Figure 51.2 depicts a supercritical bifurcation from the origin. This bifurcation is distinguished by the 
fact that the solution bifurcating from the origin at point A is stable, and occurs locally for parameter 
values t beyond the critical value (i.e., those for which the nominal equilibrium point is unstable). In 
marked difference with the situation depicted in Figure 51.1, here as the critical parameter value is crossed 
a smooth change is observed in the system operating condition. No sudden jump occurs. 

Suppose closeness of the system’s operation to the nominal equilibrium point (the origin, say) is 
a measure of the system’s performance. Then supercritical bifurcations ensure close operation to the 
nominal equilibrium, while subcritical bifurcations may lead to large excursions away from the nominal 
equilibrium point. For this reason, a supercritical bifurcation is commonly also said to be safe or soft, 
while a subcritical bifurcation is said to be dangerous or hard [32,49,52]. 
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FIGURE 51.2 Supercritical bifurcation. 


Given full information on the nominal equilibrium point, the occurrence of bifurcation is a consequence 
of the behavior of the linearized system at the equilibrium point. The manner in which the equilibrium 
point loses stability as the bifurcation parameter is varied determines the type of bifurcation that arises. 

Three types of local bifurcation and a global bifurcation are discussed next. These are, respectively, 
the stationary bifurcation, the saddle-node bifurcation, the Andronov—Hopf bifurcation, and the period 
doubling bifurcation. All of these except the saddle-node bifurcation can be safe or dangerous. However, 
the saddle-node bifurcation is always dangerous. 

There are analytical techniques for determining whether a stationary, Andronov-Hopf, or period 
doubling bifurcation is safe or dangerous. These techniques are not difficult to understand but involve 
calculations that are too lengthy to repeat here. The calculations yield formulas for so-called “bifurcation 
stability coefficients” [18], the meaning of which is addressed below. The references [1,2,4,18,28,32] can 
be consulted for details. 

What is termed here as “stationary bifurcation” is actually a special case of the usual meaning of the 
term. In the bifurcation theory literature [10], stationary bifurcation is any bifurcation of one or more 
equilibrium points from a nominal equilibrium point. When the nominal equilibrium point exists both 
before and after the critical parameter value, a stationary bifurcation “from a known solution” is said to 
occur. If the nominal solution disappears beyond the critical parameter value, a stationary bifurcation 
“from an unknown solution” is said to occur. To simplify the terminology, here the former type of 
bifurcation is called a stationary bifurcation. The saddle-node bifurcation is the most common example 
of the latter type of bifurcation. 

Andronov-Hopf bifurcation also goes by other names. “Hopf bifurcation” is the traditional name in 
the West, but this name neglects the fundamental early contributions of Andronov and his coworkers 
(see, e.g., [6]). The essence of this phenomenon was also known to Poincaré, who did not develop a 
detailed theory but used the concept in his study of lunar orbital dynamics [37, Secs. 51-52]. The same 
phenomenon is sometimes called flutter bifurcation in the engineering literature. This bifurcation of a 
limit cycle from an equilibrium point occurs when a complex conjugate pair of eigenvalues crosses the 
imaginary axis into the right half of the complex plane at jt = ;. A small-amplitude limit cycle then 
emerges from the nominal equilibrium point at |1;. 


51.3.1.2 Saddle-Node Bifurcation and Stationary Bifurcation 


Saddle-node bifurcation occurs when the linearized system has a zero eigenvalue at jt = 1; but the origin 
does not persist as an equilibrium point beyond the critical parameter value. Saddle-node bifurcation was 
discussed briefly before, and will not be discussed in any detail in the following. Several basic remarks are, 
however, in order. 


1. Saddle-node bifurcation of a nominal, stable equilibrium point entails the disappearance of the 
equilibrium upon its merging with an unstable equilibrium point at a critical parameter value. 
2. The bifurcation occurring at point B in Figure 51.1 is representative of a saddle-node bifurcation. 
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3. The nominal equilibrium point possesses a zero eigenvalue at a saddle-node bifurcation. 
4. An important feature of the saddle-node bifurcation is the disappearance, locally, of any stable 
bounded solution of the system (Equation 51.2). 


Stationary bifurcation (according to the agreed upon terminology above) is guaranteed to occur when 
a single real eigenvalue goes from being negative to being positive as | passes through the value jt-. More 
precisely, the origin of Equation 51.2 undergoes a stationary bifurcation at the critical parameter value 
. = 0 if hypotheses (S1) and (S2) hold. 


S1 Fofsystem (Equation 51.2) is sufficiently smooth in x, 1, and FH (0) = 0 forall ina neighborhood 
of 0. The Jacobian A(w) := oF (0) possesses a simple real eigenvalue (\1) such that 4(0) = 0 and 
(0) £0. 


$2 All eigenvalues of the critical Jacobian oF’ (0) besides 0 have negative real parts. 


Under (S1) and (S2), two new equilibrium points of Equation 51.2 emerge from the origin at j. = 0. 
Bifurcation stability coefficients are quantities that determine the direction of bifurcation, and in particular 
the stability of the bifurcated solutions. Next, a brief discussion of the origin and meaning of these 
quantities is given. 

Locally, the new equilibrium points occur for parameter values given by a smooth function of an 
auxiliary small parameter « (€ can be positive or negative): 


we) = Wie + pre? + pse+ (51.4) 


where the ellipsis denotes higher order terms. One of the new equilibrium points occurs for € > 0 and the 
other for € < 0. Also, the stability of the new equilibrium points is determined by the sign of an eigenvalue 
B(e) of the system linearization at the new equilibrium points. This eigenvalue is near 0 and is also given 
by a smooth function of the parameter e: 


B(e) = Bie + Boe? + Bsc? +++ (51.5) 


Stability of the bifurcated equilibrium points is determined by the sign of B(€). If B(e) < 0 the correspond- 
ing equilibrium point is stable, while if B(e) > 0 the equilibrium point is unstable. The coefficients B;, 
i= 1,2,...in the expansion above are the bifurcation stability coefficients mentioned earlier, for the case 
of stationary bifurcation. The values of these coefficients determine the local nature of the bifurcation. 

Since € can be positive or negative, it follows that if 8; 40 the bifurcation is neither subcritical 
nor supercritical. (This is equivalent to the condition 1; #0.) The bifurcation is therefore generically 
transcritical. In applications, however, special structures of system dynamics and inherent symmetries 
often result in stationary bifurcations that are not transcritical. Also, it is sometimes possible to render 
a stationary bifurcation supercritical using feedback control. For these reasons, a brief discussion of 
subcritical and supercritical pitchfork bifurcations is given next. 

IfB; = Oand B2 ¥ 0, a stationary bifurcation is known as a pitchfork bifurcation. The pitchfork bifurca- 
tion is subcritical if Bz > 0; it is supercritical if Bz < 0. The bifurcation diagram of a subcritical pitchfork 
bifurcation is depicted in Figure 51.3, and that of a supercritical pitchfork bifurcation is depicted in 
Figure 51.4. The bifurcation discussed previously for the example system (Equation 51.3) is a supercriti- 
cal pitchfork bifurcation. 


51.3.1.3 Andronov—Hopf Bifurcation 


Suppose that the origin of Equation 51.2 loses stability as the result of a complex conjugate pair of 
eigenvalues of A(t) crossing the imaginary axis. All other eigenvalues are assumed to remain stable, 
ie., their real parts are negative for all values of 1. Under this simple condition on the linearization 
of a nonlinear system, the nonlinear system typically undergoes a bifurcation. The word “typically” is 
used because there is one more condition to satisfy, but it is a mild condition. The type of bifurcation 
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FIGURE 51.3 Subcritical pitchfork bifurcation. 


that occurs under these circumstances involves the emergence of a limit cycle from the origin as wt is 
varied through |. This is the Andronov—Hopf bifurcation, a more precise description of which is given 
next. The following hypotheses are invoked in the Andronov—-Hopf Bifurcation Theorem. The critical 
parameter value is taken to be jt, = 0 without loss of generality. 


AH1 Fofsystem (Equation 51.2) is sufficiently smooth in x, 1, and F¥ (0) = 0 forall jt inaneighborhood 
of 0. The Jacobian ae (0) possesses a complex-conjugate pair of (algebraically) simple eigenvalues 
ACL) = a() + iw(), XC), such that (0) = 0, a’(0) # 0 and w, := w(0) > 0. 

AH2 All eigenvalues of the critical Jacobian oF (0) besides tiw, have negative real parts. 


The Andronov-Hopf Bifurcation Theorem asserts that, under (AH1) and (AH2), a small-amplitude 
nonconstant limit cycle (i.e., periodic solution) of Equation 51.2 emerges from the origin at 1. = 0. Locally, 
the limit cycles occur for parameter values given by a smooth and even function of the amplitude € of the 
limit cycles: 

[u(e) = pre? + get +--- (51.6) 


where the ellipsis denotes higher order terms. 

Stability of an equilibrium point of the system (Equation 51.2) can be studied using eigenvalues of the 
system linearization evaluated at the equilibrium point. The analogous quantities for consideration of 
limit cycle stability for Equation 51.2 are the characteristic multipliers of the limit cycle. (For a definition, 
see for example [10,18,28,30,32,43,46,48].) A limit cycle is stable (precisely: orbitally asymptotically stable) 
if its characteristic multipliers all have magnitude less than unity. This is analogous to the widely known 
fact that an equilibrium point is stable if the system eigenvalues evaluated there have negative real parts. 
The stability condition is sometimes stated in terms of the characteristic exponents of the limit cycle, 
quantities which are easily obtained from the characteristic multipliers. If the characteristic exponents 
have negative real parts, then the limit cycle is stable. Although it is not possible to discuss the basic theory 
of limit cycle stability in any detail here, the reader is referred to almost any text on differential equations, 
dynamical systems, or bifurcation theory for a detailed discussion (e.g., [10,18,28,30,32,43,46,48]). 


xa 


FIGURE 51.4 Supercritical pitchfork bifurcation. 
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FIGURE 51.5 Subcritical Andronov-Hopf bifurcation. 


The stability of the limit cycle resulting from an Andronov—-Hopf bifurcation is determined by the sign 
of a particular characteristic exponent B(€). This characteristic exponent is real and vanishes in the limit 
as the bifurcation point is approached. It is given by a smooth and even function of the amplitude € of the 
limit cycles: 

Ble) = Bre’ + Bae +--- (51.7) 


The coefficients jt2 and B2 in the expansions above are related by the exchange of stability formula 
Bo = —2c1'(0) p12. (51.8) 


Generically, these coefficients do not vanish. Their signs determine the direction of bifurcation. The 
coefficients B2,Bh4,... in the expansion (Equation 51.7) are the bifurcation stability coefficients for the 
case of Andronov—Hopf bifurcation. 

If B2 > 0, then locally the bifurcated limit cycle is unstable and the bifurcation is subcritical. This 
case is depicted in Figure 51.5. If Bz < 0, then locally the bifurcated limit cycle is stable (more precisely, 
one says that it is orbitally asymptotically stable [10]). This is the case of supercritical Andronov-Hopf 
bifurcation, depicted in Figure 51.6. If it happens that 82 vanishes, then stability is determined by the first 
nonvanishing bifurcation stability coefficient (if one exists). 


51.3.1.4 Period Doubling Bifurcation 


The bifurcations considered above are all local bifurcations, i.e., bifurcations from an equilibrium point 
of the system (Equation 51.2). Solutions emerging at these bifurcation points can themselves undergo 
further bifurcations. A particularly important scenario involves a global bifurcation known as the period 
doubling bifurcation. 

To describe the period doubling bifurcation, consider the one-parameter family of nonlinear systems 
(Equation 51.2). Suppose that Equation 51.2 has a limit cycle y" for a range of values of the real parameter 


YA 
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FIGURE 51.6 Supercritical Andronov-Hopf bifurcation. 
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H<H, 


FIGURE 51.7 Period doubling bifurcation (supercritical case). 


1. Moreover, suppose that for all values of 1 to one side (say, less than) a critical value |1;, all the 
characteristic multipliers of y" have magnitude less than unity. If exactly one characteristic multiplier 
exits the unit circle at [1 = (1-, and does so at the point (—1, 0), and if this crossing occurs with a nonzero 
rate with respect to 1, then one can show that a period doubling bifurcation from y" occurs at p= [c. 
(See, e.g., [4] and references therein.) 

This means that another limit cycle, initially of twice the period of y"*, emerges from y" at jt = Wc. 
Typically, the bifurcation is either supercritical or subcritical. In the supercritical case, the period doubled 
limit cycle is stable and occurs for parameter values on the side of \1, for which the limit cycle y" is 
unstable. In the subcritical case, the period doubled limit cycle is unstable and occurs on the side of |, 
for which the limit cycle y" is stable. In either case, an exchange of stability is said to have occurred 
between the nominal limit cycle y" and the bifurcated limit cycle. Figure 51.7 depicts a supercritical 
period doubling bifurcation. In this figure, a solid curve represents a stable limit cycle, while a dashed 
curve represents an unstable limit cycle. The figure assumes that the nominal limit cycle loses stability as 
|. increases through [1¢. 

The direction ofa period doubling bifurcation can easily be determined in discrete-time, using formulas 
that have been derived in the literature (see, e.g., [4]). Recently, an approximate test that applies in 
continuous-time has been derived using the harmonic balance approach [47]. 


51.3.2 Chaos 


Bifurcations of equilibrium points and limit cycles are well understood and there is little room for 
alternative definitions of the main concepts. Although the notation, style, and emphasis may differ 
among various presentations, the main concepts and results stay the same. Unfortunately, the situ- 
ation is not quite as tidy in regard to discussions of chaos. There are several distinct definitions of 
chaotic behavior of dynamical systems. There are also some aspects of chaotic motion that have been 
found to be true for many systems but have not been proved in general. The aim of this subsection 
is to summarize in a nonrigorous fashion some important aspects of chaos that are widely agreed 
upon. 

The following working definition of chaos will suffice for the purposes of this chapter. The definition 
is motivated by [46, p. 323] and [11, p. 50]. It uses the notion of “attractor” defined in section two, and 
includes the definition of an additional notion, namely that of “strange attractor.” 


A solution trajectory of a deterministic system (such as Equation 51.1) is chaotic if it converges to a 
strange attractor. A strange attractor is a bounded attractor that: (1) exhibits sensitive dependence 
on initial conditions, and (2) cannot be decomposed into two invariant subsets contained in disjoint 
open sets. 


A few remarks on this working definition are in order. An aperiodic motion is one that is not 
periodic. Long-term behavior refers to steady-state behavior, i-e., system behavior that persists after 
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FIGURE 51.8 Strange attractor with embedded unstable limit cycle. 


the transient decays. Sensitive dependence on initial conditions means that for almost any initial 
condition lying on the strange attractor, there exists another initial condition as close as desired to 
the given one such that the solution trajectories starting from these two initial conditions separate by 
at least some prespecified amount after some time. The requirement of nondecomposability simply 
ensures that strange attractors are considered as being distinct if they are not connected by any system 
trajectories. 

Often, sensitive dependence on initial conditions is defined in terms of the presence of at least one pos- 
itive “Lyapunov exponent” (e.g., [34,35]). Further discussion of this viewpoint would entail technicalities 
that are not needed in the sequel. 

From a practical point of view, a chaotic motion can be defined as a bounded invariant motion 
of a deterministic system that is not an equilibrium solution or a periodic solution or a quasiperiodic 
solution [32, p. 277]. (A quasiperiodic function is one that is composed of finitely many periodic functions 
with incommensurate frequencies. See [32, p. 231].) 

There are two more important aspects of strange attractors and chaos that should be noted, since they 
play an important role in a technique for control of chaos discussed in Section 51.7. These are: 


1. A strange attractor generally has embedded within itself infinitely many unstable limit cycles. For 
example, Figure 51.8a depicts a strange attractor, and Figure 51.8b depicts an unstable limit cycle 
that is embedded in the strange attractor. Note that the shape of the limit cycle resembles that of 
the strange attractor. This is to be expected in general. The limit cycle chosen in the plot happens 
to be one of low period. 

2. The trajectory starting from any point on a strange attractor will, after sufficient time, pass as 
close as desired to any other point of interest on the strange attractor. This follows from the 
indecomposability of strange attractors noted previously. 


An important way in which chaotic behavior arises is through sequences of bifurcations. A well- 
known such mechanism is the period doubling route to chaos, which involves the following sequence of 
events: 


1. A stable limit cycle loses stability, and a new stable limit cycle of double the period emerges. 
(The original stable limit cycle might have emerged from an equilibrium point via a supercritical 
Andronov-Hopf bifurcation.) 

2. The new stable limit cycle loses stability, releasing another stable limit cycle of twice its period. 

3. There is a cascade of such events, with the parameter separation between each two successive 
events decreasing geometrically.* This cascade culminates in a sustained chaotic motion (a strange 
attractor). 


* This is true exactly in an asymptotic sense. The ratio in the geometric sequence is a universal constant discovered by 


Feigenbaum. See, e.g., [11,32,34,43,46,48]. 
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51.4 Bifurcations and Chaos in Physical Systems 


In this brief section, a list of representative physical systems that exhibit bifurcations and/or chaotic 
behavior is given. The purpose is to provide practical motivation for study of these phenomena and their 
control. 

Examples of physical systems exhibiting bifurcations and/or chaotic behavior include the following. 


« Anaircraft stalls for flight under a critical speed or above a critical angle-of-attack (e.g., [8,13,39,51]). 

« Aspects of laser operation can be viewed in terms of bifurcations. The simplest such observation 
is that a laser can only generate significant output if the pump energy exceeds a certain threshold 
(e.g., [19,46]). More interestingly, as the pump energy increases, the laser operating condition can 
exhibit bifurcations leading to chaos (e.g., [17]). 

¢ The dynamics of ships at sea can exhibit bifurcations for wave frequencies close to a natural 
frequency of the ship. This can lead to large-amplitude oscillations, chaotic motion, and ship 
capsize (e.g., [24,40]). 

« Lightweight, flexible, aircraft wings tend to experience flutter (structural oscillations) (e.g., [39]) 
(along with loss of control surface effectiveness (e.g., [15])). 

« At peaks in customer demand for electric power (such as during extremes in weather), the sta- 
bility margin of an electric power network may become negligible, and nonlinear oscillations or 
divergence (“voltage collapse”) can occur (e.g., [12]). 

¢ Operation of aeroengine compressors at maximum pressure rise implies reduced weight require- 
ments but also increases the risk for compressor stall (e.g., [16,20,25]). 

« Asimple model for the weather consists of fluid in a container (the atmosphere) heated from below 
(the sun’s rays reflected from the ground) and cooled from above (outer space). A mathematical 
description of this model used by Lorenz [27] exhibits bifurcations of convective and chaotic 
solutions. This has implications also for boiling channels relevant to heat-exchangers, refrigeration 
systems, and boiling water reactors [14]. 

¢ Bifurcations and chaos have been observed and studied in a variety of chemically reacting systems 
(e.g., [42]). 

¢ Population models useful in the study and formulation of harvesting strategies exhibit bifurcations 
and chaos (e.g., [19,31,46]). 


51.5 Control of Parametrized Families of Systems 


Tracking of a desired trajectory (referred to as regulation when the trajectory is an equilibrium), is a 
standard goal in control system design [26]. In applying linear control system design to this standard 
problem, an evolving (nonstationary) system is modeled by a parametrized family of time-invariant 
(stationary) systems. This approach is at the heart of the gain scheduling method, for example [26]. 

In this section, some basic concepts in control of parameterized families of systems are reviewed, and 
a notion of stressed operation is introduced. Control laws for nonlinear systems usually consist of a 
feedforward control plus a feedback control. The feedforward part of the control is selected first, followed 
by the feedback part. This decomposition of control laws is discussed in the next subsection, and will 
be useful in the discussions of control of bifurcations and chaos. In the second subsection, a notion of 
“stressed operation” of a system is introduced. Stressed systems are not the only ones for which control 
of bifurcations and chaos are relevant, but they are an important class for which such control issues need 
to be evaluated. 
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51.5.1 Feedforward/Feedback Structure of Control Laws 


Control designs for nonlinear system can usually be viewed as proceeding in two main steps [26, Chapters 
2, 14], [45, Chapter 3]: 


1. Feedforward control 
2. Feedback control 


In Step 1, the feedforward part of the control input is selected. Its purpose is to achieve a desired candidate 
operating condition for the system. If the system is considered as depending on one or more parameters, 
then the feedforward control will also depend on the parameters. The desired operating condition can 
often be viewed as an equilibrium point of the nonlinear system that varies as the system parameters 
are varied. There are many situations when this operating condition is better viewed as a limit cycle that 
varies with the parameters. In Step 2, an additional part of the control input is designed to achieve desired 
qualities of the transient response in a neighborhood of the nominal operating condition. Typically this 
second part of the control is selected in feedback form. 

In the following, the feedforward part of the control input is taken to be designed already and reflected 
in the system dynamical equations. The discussion centers on design of the feedback part of the control 
input. Because of this, it is convenient to denote the feedback part of the control simply by u and to view 
the feedforward part as being determined a priori. It is also convenient to take u to be small (close to zero) 
near the nominal operating condition. (Any offset in u is considered part of the feedforward control.) 

It is convenient to view the system as depending on a single exogeneous parameter, denoted by 1. 
For instance, ,t can represent the set-point of an aircraft’s angle-of-attack, or the power demanded of an 
electric utility by a customer. In the former example, the control u might denote actuation of the aircraft’s 
elevator angles about the nominal settings. In the latter example, u can represent a control signal in the 
voltage regulator of a power generator. 

Consider, then, a nonlinear system depending on a single parameter | 


x =f" (x, u). (51.9) 


Here x € R” is the n-dimensional state vector, u is the feedback part of the control input, and jt is a 
parameter. Both u and yw are taken to be scalar-valued for simplicity. The dependence of the system 
equations on x, u, and i. is assumed smooth; i.e., f is jointly continuous and several times differentiable 
in these variables. This system is thus actually a one-parameter family of nonlinear control systems. The 
parameter |1 is viewed as being allowed to change so slowly that its variation can be taken as quasistatic. 

Suppose that the nominal operating condition of the system is an equilibrium point x°(\1) that depends 
on the parameter |1. For simplicity of notation, suppose that the state x has been chosen so that this 
nominal equilibrium point is the origin for the range of values of \1 for which it exists (x°(\1) = 0). Recall 
that the nominal equilibrium is achieved using feedforward control. Although the process of choosing a 
feedforward control is not elaborated on here, it is important to emphasize that in general this process 
aims at securing a particular desired candidate operating condition. The feedforward control thus is 
expected to result in an acceptable form for the nominal operating condition, but there is no reason to 
expect that other operating conditions will also behave in a desirable fashion. As the parameter varies, 
the nominal operating condition may interact with other candidate operating conditions in bifurcations. 
The design of the feedback part of the control should take into account the other candidate operating 
conditions in addition to the nominal one. 

For simplicity, suppose the control u is not allowed to introduce additional dynamics. That is, suppose 
u is required to be a direct state feedback (or static feedback), and denote it by u = u(x, ). Note that in 
general u can depend on the parameter jt. That is, it can be scheduled. Since in the discussion above it 
was assumed that u is small near the nominal operating condition, it is assumed that u(0, 1) = 0 (for the 
parameter range in which the origin is an equilibrium). 
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Denote the linearization of Equation 51.9 at x° = 0, u = 0 by 


x = A()x + b(w)u. (51.10) 
Here, 
1 
Aw) = 2.0 
Ma 
and 
a ie 
b(w) = “au 0). 


Consider how control design for the linearized system depends on the parameter 1. Recall the termi- 
nology from linear system theory that the pair (A(t), b(|1)) is controllable if the system (Equation 51.10) 
is controllable. Recall also that there are several simple tests for controllability, one of which is that the 
controllability matrix 


(b(), A(u)b(2), (A(L))2B(), -- - (A() PBL) 


is of full rank. (In this case this is equivalent to the matrix being nonsingular, since here the controllability 
matrix is square.) 

If is such that the pair (A(t), b(\)) is controllable, then a standard linear systems result asserts the 
existence of a linear feedback u(x, ~) = —k(w)x stabilizing the system. (The associated closed-loop system 
would be x = (A(w) — b(\)k(\t))x.) Stabilizability tests not requiring controllability also exist, and these 
are more relevant to the problem at hand. Even more interesting from a practical perspective is the issue 
of output feedback stabilizability, since not all state variables are accessible for real-time measurement in 
many systems. As \1 is varied over the desired regime of operability, the system (Equation 51.10) may lose 
stability and stabilizability. 


51.5.2 Stressed Operation and Break-Down of Linear Techniques 


A main motivation for the study of control of bifurcations is the need in some situations to operate a 
system in a condition for which the stability margin is small and linear (state or output) feedback is 
ineffective as a means for increasing the stability margin. Such a system is sometimes referred to as being 
“pushed to its limits,” or “heavily loaded.” In such situations, the ability of the system to function in a state 
of increased loading is a measure of system performance. Thus, the link between increased loading and 
reduced stability margin can be viewed as a performance vs. stability trade-off. Systems operating with a 
reduced achievable margin of stability may be viewed as being “stressed.” This trade-off is not a general 
fact that can be proved in a rigorous fashion, but has been found to occur in a variety of applications. 
Examples of this trade-off are given at the end of this subsection. 

Consider a system that is weakly damped and for which the available controls cannot compensate 
with sufficient additional damping. Such a situation may arise for a system for some ranges of parameter 
values and not for others. Let the operating envelope of a system be the possible combinations of system 
parameters for which system operability is being considered. Linear control system methods lose their 
effectiveness on that part of the operating envelope for which the operating condition of interest of system 
(Equation 51.10) is either: 


1. not stabilizable with linear feedback using available sensors and actuators 

2. linearly stabilizable using available sensors and actuators but only with unacceptably high feedback 
gains 

3. vulnerable in the sense that small parameter changes can destroy the operating condition completely 
(as in a saddle-node bifurcation) 
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Operation in this part of the desired operating envelope can be referred to by terms such as “stressed 
operation.” 

An example of the trade-off noted previously is provided by an electric power system under conditions 
of heavy loading. At peaks in customer demand for electric power (such as during extremes in weather), 
the stability margin may become negligible, and nonlinear dynamical behaviors or divergence may arise 
(e.g., [12]). The divergence, known as voltage collapse, can lead to system blackout. Another example 
arises in operation of an aeroengine compressor at its peak pressure rise. The increased pressure rise 
comes at the price of nearness to instability. The unstable modes that can arise are strongly related to 
flow asymmetry modes that are unstabilizable by linear feedback to the compression system throttle. 
However, bifurcation control techniques have yielded valuable nonlinear throttle actuation techniques 
that facilitate operation in these circumstances with reduced risk of stall [16,25]. 


51.6 Control of Systems Exhibiting Bifurcation Behavior 


Most engineering systems are designed to operate with a comfortable margin of stability. This means 
that disturbances or moderate changes in system parameters are unlikely to result in loss of stability. 
For example, a jet airplane in straight level flight under autopilot control is designed to have a large 
stability margin. However, engineering systems with a usually comfortable stability margin may at times 
be operated at a reduced stability margin. A jet airplane being maneuvered at high angle-of-attack to gain 
an advantage over an enemy aircraft, for instance, may have a significantly reduced stability margin. Ifa 
system operating condition actually loses stability as a parameter (like angle-of-attack) is slowly varied, 
then generally it is the case that a bifurcation occurs. This means that at least one new candidate operating 
condition emerges from the nominal one at the point of loss of stability. The purpose of this section 
is to summarize some results on control of bifurcations, with an emphasis placed on control of local 
bifurcations. Control of a particular global bifurcation, the period doubling bifurcation, is considered in 
the next section on control of chaos. This is because control of a period doubling bifurcation can result 
in quenching of the period doubling route to chaos summarized at the end of section three. 

Bifurcation control involves designing a control input for a system to result in a desired modification to 
the system’s bifurcation behavior. In Section 51.5, the division of control into a feedforward component 
and a feedback component was discussed. Both components of a control law can be viewed in terms of 
bifurcation control. The feedforward part of the control sets the equilibrium points of the system, and may 
influence the stability margin as well. The feedback part of the control has many functions, one of which 
is to ensure adequate stability of the desired operating condition over the desired operating envelope. 
Linear feedback is used to ensure an adequate margin of stability over a desired parameter range. Use 
of linear feedback to “delay” the onset of instability to parameter ranges outside the desired operating 
range is a common practice in control system design. An example is the gain scheduling technique [26]. 
Delaying instability modifies the bifurcation diagram of a system. Often, the available control authority 
does not allow stabilization of the nominal operating condition beyond some critical parameter value. At 
this value, instability leads to bifurcations of new candidate operating conditions. For simplicity, suppose 
that a single candidate operating condition is born at the bifurcation point. Another important goal 
in bifurcation control is to ensure that the bifurcation is supercritical (i.e., safe) and that the resulting 
candidate operating condition remains stable and close to the original operating condition for a range 
of parameter values beyond the critical value. The need for control laws that soften (stabilize) a hard 
(unstable) bifurcation has been discussed earlier in this chapter. This need is greatest in stressed systems, 
since in such systems delay of the bifurcation by linear feedback is not viable. A soft bifurcation provides 
the possibility of an alternative operating condition beyond the regime of operability at the nominal 
condition. 

Both of these goals (delaying and stabilization) basically involve local considerations and can be 
approached analytically (if a good system model is available). Another goal might entail a reduction 
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in amplitude of any bifurcated solutions over some prespecified parameter range. This goal is generally 
impossible to work with on a completely analytical basis. It requires extensive numerical study in addition 
to local analysis near the bifurcation(s). 

In the most severe local bifurcations (saddle-node bifurcations), neither the nominal equilibrium 
point nor any bifurcated solution exists past the bifurcation. Even in such cases, an understanding of 
bifurcations provides some insight into control design for safe operation. For example, it may be possible 
to use this understanding to determine (or introduce via added control) a warning signal that becomes 
more pronounced as the severe bifurcation is approached. This signal would alert the high-level control 
system (possibly a human operator) that action is necessary to avoid catastrophe. 

In this section, generally it is assumed that the feedforward component of the control has been pre- 
determined, and the goal is to design the feedback component. An exception is the following brief 
discussion of a real-world example of the use of feedforward control to modify a system’s operating 
condition and its stability margin in the face of large parameter variations. In short, this is an example 
where feedforward controls are used to successfully avoid the possibility of bifurcation. During takeoff and 
landing of a commercial jet aircraft, one can observe the deployment of movable surfaces on the leading- 
and trailing-edges of the wings. These movable surfaces, called flaps and slats, or camber changers [13], 
result in a nonlinear change in the aerodynamics, and, in turn, in an increased lift coefficient [13,51]. 
This is needed to allow takeoff and landing at reduced speeds. Use of these surfaces has the drawback of 
reducing the critical angle-of-attack for stall, resulting in a reduced stability margin. A common method 
to alleviate this effect is to incorporate other devices, called vortex generators. These are small airfoil- 
shaped vanes, protruding upward from the wings [13]. The incorporation of vortex generators results in 
a further modification to the aeodynamics, moving the stall angle-of-attack to a higher value. Use of the 
camber changers and the vortex generators are examples of feedforward control to modify the operating 
condition within a part of the aircraft’s operating envelope. References [13] and [51] provide further 
details, as well as diagrams showing how the lift coefficient curve is affected by these devices. 


51.6.1 Local Direct State Feedback 


To give a flavor of the analytical results available in the design of the feedback component in bifurcation 
control, consider the nonlinear control system (Equation 51.9), repeated here for convenience: 


x=f" (x, u). (51.11) 


Here, u represents the feedback part of the control law; the feedforward part is assumed to be incorpo- 
rated into the function f. The technique and results of [1,2] form the basis for the following discussion. 
Details are not provided since they would require introduction of considerable notation related to multi- 
variate Taylor series. However, an illustrative example is given based on formulas available in [1,2]. 

Suppose for simplicity that Equation 51.11 with u =0 possesses an equilibrium at the origin for 
a parameter range of interest (including the value 1 =0). Moreover, suppose that the origin of 
Equation 51.11 with the control set to zero undergoes either a subcritical stationary bifurcation or a 
subcritical Andronov—-Hopf bifurcation at the critical parameter value 1. = 0. Feedback control laws of 
the form u = u(x) (“static state feedbacks”) are derived in [1,2] that render the bifurcation supercritical. 

For the Andronov-Hopf bifurcation, this is achieved using a formula for the coefficient B2 in the 
expansion (Equation 51.7) of the characteristic exponent for the bifurcated limit cycle. Smooth nonlinear 
controls rendering 82 < 0 are derived. For the stationary bifurcation, the controlled system is desired to 
display a supercritical pitchfork bifurcation. This is achieved using formulas for the coefficients B; and B2 
in the expansion (Equation 51.5) for the eigenvalue of the bifurcated equilibrium determining stability. 
Supercriticality is insured by determining conditions on u(x) such that B; = 0 and £2 < 0. 

The following example is meant to illustrate the technique of [2]. The calculations involve use of for- 
mulas from [2] for the bifurcation stability coefficients B; and Bz in the analysis of stationary bifurcations. 
The general formulas are not repeated here. 
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Consider the one-parameter family of nonlinear control systems 


dy = xy +2 + x9 +9, (51.12) 


ky = —X2 x1X3 + LuU+ xX] U. (51.13) 


Here x1, x2 are scalar state variables, and 1 is a real-valued parameter. This is meant to represent a 
system after application of a feedforward control, so that u is to be designed in feedback form. The nominal 
operating condition is taken to be the origin (x1, x2) = (0,0), which is an equilibrium of (Equations 51.12 
and 51.13) when the control u = 0 for all values of the parameter . 

Local stability analysis at the origin proceeds in the standard manner. The Jacobian matrix A(\1) of the 
right side of (Equations 51.12 and 51.13) is given by 


A(p) = G a) (51.14) 


The system eigenvalues are , and —1. Thus, the origin is stable for |. < 0 but is unstable for jt > 0. The 
critical value of the bifurcation parameter is therefore |1, = 0. Since the origin persists as an equilibrium 
point past the bifurcation, and since the critical eigenvalue is 0 (not an imaginary pair), it is expected that 
a stationary bifurcation occurs. The stationary bifurcation that occurs for the open-loop system can be 
studied by solving the pair of algebraic equations 


O= px +x2+21x3 +23, (51.15) 
0 = —x2 — xx} (51.16) 


for a nontrivial (i.e., nonzero) equilibrium (x!, x?) near the origin for j near 0. Adding Equation 51.15 to 
Equation 51.16 gives 
O=px +x}. (51.17) 


Disregarding the origin, this gives two new equilibrium points that exist for jt < 0, namely 


x(u) = Garr (51.18) 


Since these bifurcated equilibria occur for parameter values (\1 < 0) for which the nominal operating 
condition is unstable, the bifurcation is a subcritical pitchfork bifurcation. 

The first issue addressed in the control design is the possibility of using linear feedback to delay the 
bifurcation to some positive value of . This would require stabilization of the origin at 1 = 0. Because of 
the way in which the control enters the system dynamics, however, the system eigenvalues are not affected 
by linear feedback at the parameter value . = 0. To see this, simply note that in Equation 51.13, the term 
ju vanishes when 1 = 0, and the remaining impact of the control is through the term x, u. This latter 
term would result only in the addition of quadratic terms to the right side of Equations 51.12 and 51.13 
for any linear feedback u. Hence, the system is an example of a stressed nonlinear system for | near 0. 

Since the pitchfork bifurcation cannot be delayed by linear feedback, next consider the possibility of 
rendering the pitchfork bifurcation supercritical using nonlinear feedback. This rests on determining how 
feedback affects the bifurcation stability coefficients B; and B2 for this stationary bifurcation. Once this is 
known, it is straightforward to seek a feedback that renders 8B; = 0 and B2 < 0. The formulas for B; and 
Bz derived in [2] simplify for systems with no quadratic terms in the state variables. For such systems, 
the coefficient B always vanishes, and the calculation of 2 also simplifies. Since the dynamical equations 
51.12 and 51.13 in the example contain no quadratic terms in x, it follows that Bj =0 in the absence 
of control. Moreover, if the control contains no linear terms, then it will not introduce quadratic terms 
into the dynamics. Hence, for any smooth feedback u(x) containing no linear terms in x, the bifurcation 
stability coefficient B; = 0. 
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As for the bifurcation stability coefficient B2, the pertinent formula in [2] applied to the open-loop 
system yields B2 = 2. Thus, a subcritical pitchfork bifurcation is predicted for the open-loop system, a 
fact that was established above using simple algebra. Now let the control consist of a quadratic function 
of the state and determine conditions under which B2 < 0 for the closed-loop system.* Thus, consider u 
to be of the form 


u(x), X2) = kyx? kx x2 k3x}. (51.19) 


The formula in [2] yields that Bz for the closed-loop system is then given by 
Bo = 2(1— ky). (51.20) 


Thus, to render the pitchfork bifurcation in Equations 51.12 and 51.13 supercritical, it suffices to take 
u to be the simple quadratic function 
u(x1, x2) = —kyx7, (51.21) 


with any gain k; > 1. In fact, other quadratic terms, as well as any other cubic or higher order terms, 
can be included along with this term without changing the local result that the bifurcation is rendered 
supercritical. Additional terms can be useful in improving system behavior as the parameter leaves a local 
neighborhood of its critical value. 


51.6.2 Local Dynamic State Feedback 


Use of a static state feedback control law u = u(x) has potential disadvantages in nonlinear control of 
systems exhibiting bifurcation behavior. To explain this, consider the case of an equilibrium x°({1) as the 
nominal operating condition. The equilibrium is not translated to the origin to illustrate how it is affected 
by feedback. In general, a static state feedback 


u= u(x —x°(W)) (51.22) 


designed with reference to the nominal equilibrium path x°(\) of Equation 51.11 will affect not only 
the stability of this equilibrium but also the location and stability of other equilibria. Now suppose that 
Equation 51.11 is only an approximate model for the physical system of interest. Then the nominal 
equilibrium branch will also be altered by the feedback. A main disadvantage of such an effect is the 
wasted control energy that is associated with the forced alteration of the system equilibrium structure. 
Other disadvantages are that system performance is often degraded by operating at an equilibrium that 
differs from the one at which the system is designed to operate. Moreover, by influencing the locations of 
system equilibria, the feedback control is competing with the feedforward part of the control. 

For these reasons, dynamic state feedback-type bifurcation control laws have been developed that do 
not affect the locations of system equilibria [22,23,50]. The method involves incorporation of filters called 
“washout filters” into the controller architecture. A washout filter-aided control law preserves all system 
equilibria, and does so without the need for an accurate system model. 

A washout filter is a stable high pass filter with zero static gain [15, p. 474]. The typical transfer function 
for a washout filter is 
vis) ss 


OO a ea 


(51.23) 


where x; is the input variable to the filter and y; is the output of the filter. A washout filter produces a 
nonzero output only during the transient period. Thus, such a filter “washes out” sensed signals that have 
settled to constant steady-state values. Washout filters occur in control systems for power systems [5, p. 
277], [41, Chapter 9] and aircraft [7,15,39,45]. 


* Cubic terms are not included because they would result in quartic terms on the right side of Equations 51.12 and 51.13, 


while the formula for 2 in [2] involves only terms up to cubic order in the state. 
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Washout filters are positioned in a control system so that a sensed signal being fed back to an actuator 
first passes through the washout filter. If, due to parameter drift or intentional parameter variation, the 
sensed signal has a steady-state value that deviates from the assumed value, the washout filter will give 
a zero output and the deviation will not propagate. If a direct state feedback were used instead, the 
steady-state deviation in the sensed signal would result in the control modifying the steady-state values 
of the other controlled variables. As an example, washout filters are used in aircraft yaw damping control 
systems to prevent these dampers from “fighting” the pilot in steady turns [39, p. 947]. 

From a nonlinear systems perspective, the property of washout filters described above translates to 
achieving equilibrium preservation, i.e., zero steady-state tracking error, in the presence of system uncer- 
tainties. 

Washout filters can be incorporated into bifurcation control laws for Equation 51.11. This should be 
done only after the feedforward control has been designed and incorporated and part of the feedback 
control ensuring satisfactory equilibrium point structure has also been designed and incorporated into 
the dynamics. Otherwise, since washout filters preserve equilibrium points, there will be no possibility of 
modifying the equilibria. It is assumed below that these two parts of the control have been chosen and 
implemented. 

For each system state variable x;,i=1,...,n, in Equation 51.11, introduce a washout filter governed 
by the dynamic equation 


Zi = xj — dizi (51.24) 


along with output equation 


Vix Xj — diZj. (51.25) 


Here, the dj are positive parameters (this corresponds to using stable washout filters). Finally, require the 
control u to depend only on the measured variables y, and that u(y) satisfy u(0) = 0. 

In this formulation, n washout filters, one for each system state, are present. In fact, the actual number 
of washout filters needed, and hence also the resulting increase in system order, can usually be taken less 
than n. 

It is straightforward to see that washout filters result in equilibrium preservation and automatic equilib- 
rium (operating point) following. Indeed, since u(0) = 0, it is clear that y vanishes at steady-state. Hence, 
the x subvector of a closed-loop equilibrium point (x, z) agrees exactly with the open-loop equilibrium 
value of x. Also, since y; may be re-expressed as 


yi = Xj — diz; = (x1 — xP (W)) — (zi — 2? (W)), (51.26) 


the control function u = u(y) is guaranteed to center at the correct operating point. 


51.7 Control of Chaos 


Chaotic behavior of a physical system can either be desirable or undesirable, depending on the application. 
It can be beneficial for many reasons, such as enhanced mixing of chemical reactants, or, as proposed 
recently [36], as a replacement for random noise as a masking signal in a communication system. Chaos 
can, on the other hand, entail large amplitude motions and oscillations that might lead to system failure. 
The control techniques discussed in this section have as their goal the replacement of chaotic behavior by 
a nonchaotic steady-state behavior. The first technique discussed is that proposed by Ott, Grebogi, and 
Yorke [33]. The Ott-Grebogi-Yorke (OGY) method sparked significant interest and activity in control 
of chaos. The second technique discussed is the use of bifurcation control to delay or extinguish the 
appearance of chaos in a family of systems. 
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51.7.1 Exploitation of Chaos for Control 


Ott, Grebogi, and Yorke [33] proposed an approach to control of chaotic systems that involves use of 
small controls and exploitation of chaotic behavior. To explain this method, recall from section three 
that a strange attractor has embedded within itself a “dense” set (infinitely many) of unstable limit cycles. 
The strange attractor is a candidate operating condition, according to the definition in section two. In 
the absence of control, it is the actual system operating condition. Suppose that system performance 
would be significantly improved by operation at one of the unstable limit cycles embedded in the strange 
attractor. The OGY method replaces the originally chaotic system operation with operation along the 
selected unstable limit cycle. 

Figure 51.8 depicts a strange attractor along with a particular unstable limit cycle embedded in it. It is 
helpful to keep such a figure in mind in contemplating the OGY method. The goal is to reach a limit cycle 
such as the one shown in Figure 51.8, or another of some other period or amplitude. Imagine a trajectory 
that lies on the strange attractor in Figure 51.8, and suppose the desire is to use control to force the tra- 
jectory to reach the unstable limit cycle depicted in the figure and to remain there for all subsequent time. 

Control design to result in operation at the desired limit cycle is achieved by the following reasoning. 
First, note that the desired unstable limit cycle is also a candidate operating condition. Next, recall from 
section three that a trajectory on the strange attractor will, after sufficient time, pass as close as desired 
to any other point of interest on the attractor. Thus, the trajectory will eventually come close (indeed, 
arbitrarily close) to the desired limit cycle. Thus, no control effort whatsoever is needed in order for the 
trajectory to reach the desired limit cycle—chaos guarantees that it will. To maintain the system state 
on the desired limit cycle, a small stabilizing control signal is applied once the trajectory enters a small 
neighborhood of the limit cycle. Since the limit cycle is rendered stable by this control, the trajectory will 
converge to the limit cycle. If noise drives the trajectory out of the neighborhood where control is applied, 
the trajectory will wander through the strange attractor again until it once again enters the neighborhood 
and remains there by virtue of the control. 

Note that the control obtained by this method is an example of a variable structure control, since it is 
“on” in one region in state space and “off” in the rest of the space. Also, the particular locally stabilizing 
control used in the neighborhood of the desired limit cycle has not been discussed in the foregoing. 
This is because several approaches are possible, among them pole placement [38]. See [34,35] for further 
discussion. 

Two particularly significant strengths of the OGY technique are: 


1. It requires only small controls 
2. It can be applied to experimental systems for which no mathematical model is available 


The first of these strengths is due to the assumption that operation at an unstable limit cycle embedded 
in the strange attractor is desirable. If none of the embedded limit cycles provides adequate performance, 
then a large control could possibly be used to introduce a new desirable candidate operating condition 
within the strange attractor. This could be followed by a small control possibly designed within the 
OGY framework. Note that the large control would be a feedforward control, in the terminology used 
previously in this chapter. For a discussion of the second main strength of the OGY control method 
mentioned above, see, e.g., [17,33,34]. It suffices to note here that a construction known as experimental 
delay coordinate embedding is one means to implement this control method without a priori knowledge 
of a reliable mathematical model. 

Several interesting applications of the OGY control method have been performed, including control of 
cardiac chaos (see [35]). In [17], a multimode laser was controlled well into its usually unstable regime. 


51.7.2 Bifurcation Control of Routes to Chaos 


The bifurcation control techniques discussed in section six have direct relevance for issues of control 
of chaotic behavior of dynamical systems. This is because chaotic behavior often arises as a result of 
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bifurcations, such as through the period doubling route to chaos. The bifurcation control technique 
discussed earlier in this chapter is model-based. Thus, the control of chaos applications of the technique 
also require availability of a reliable mathematical model. 

Only a few comments are given here on bifurcation control of routes to chaos, since the main tool has 
already been discussed in section six. The cited references may be consulted for details and examples. 

In [50], a thermal convection loop model is considered. The model, which is equivalent to the Lorenz 
equations mentioned in Section 51.4, displays a series of bifurcations leading to chaos. In [50], an 
Andronov-Hopf bifurcation that occurs in the model is rendered supercritical using local dynamic state 
feedback (of the type discussed in Section 51.6). The feedback is designed using local calculations at the 
equilibrium point of interest. It is found that this simple control law results in elimination of the chaotic 
behavior in the system. From a practical perspective, this allows operation of the convection loop in a 
steady convective state with a desired velocity and temperature profile. 

Feedback control to render supercritical a previously subcritical period doubling bifurcation was 
studied in [4,47]. In [4], a discrete-time model is assumed, whereas a continuous-time model is assumed 
in [47]. The discrete-time model used in [4] takes the form (k is an integer) 


x(k +1) =f! (x(k), u(k)) (51.27) 


where x(k) € R” is the state, u(k) is a scalar control input,  € R is the bifurcation parameter, and the 
mapping f" is sufficiently smooth in x, u, and w. The continuous-time model used in [47] is identical to 
Equation 51.9. 

For discrete-time systems, a limit cycle must have an integer period. A period-1 limit cycle sheds a 
period-2 limit cycle upon period doubling bifurcation. The simplicity of the discrete-time setting results 
in explicit formulas for bifurcation stability coefficients and feedback controls [4]. By improving the 
stability characteristics of a bifurcated period-2 limit cycle, an existing period doubling route to chaos can 
be extinguished. Moreover, the period-2 limit cycle will then remain close to the period-1 limit cycle for 
an increased range of parameters. 

For continuous-time systems, limit cycles cannot in general be obtained analytically. Thus, [47] employs 
an approximate analysis technique known as harmonic balance. Approximate bifurcation stability coef- 
ficients are obtained, and control to delay the onset of period doubling bifurcation or stabilize such a 
bifurcation is discussed. 

The washout filter concept discussed in section six is extended in [4] to discrete-time systems. In [47], 
an extension of the washout filter concept is used that allows approximate preservation of limit cycles of 
a certain frequency. 


51.8 Concluding Remarks 


Control of bifurcations and chaos is a developing area with many interesting avenues for research and 
for application. Some of the tools and ideas that have been used in this area were discussed. Connections 
among these concepts, and relationships to traditional control ideas, have been emphasized. 
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Further Reading 


Detailed discussions of bifurcation and chaos are available in many excellent books (e.g., [6,10,11,18,19, 
28,30,32,34,35,43,46,48]). These books also discuss a variety of interesting applications. Many examples of 
bifurcations in mechanical systems are given in [52]. There are also several journals devoted to bifurcations 
and chaos. Of particular relevance to engineers are Nonlinear Dynamics, the International Journal of 
Bifurcation and Chaos, and the journal Chaos, Solitons and Fractals. 

The book [45] discusses feedforward control in the context of “trimming” an aircraft using its nonlinear 
equations of motion and the available controls. Bifurcation and chaos in flight dynamics are discussed 
in [8]. Lucid explanations on specific uses of washout filters in aircraft control systems are given in [15, pp. 
474-475], [7, pp. 144-146], [39, pp. 946-948 and pp. 1087-1095], and [45, pp. 243-246 and p. 276]. The 
book [31] discussed applications of nonlinear dynamics in biology and population dynamics. 
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Computational issues related to bifurcation analysis are addressed in [43]. Classification of bifurcations 
as safe or dangerous is discussed in [32,43,48,49]. 

The edited book [14] contains interesting articles on research needs in applications of bifurcations and 
chaos. 

The article [3] contains a large number of references on bifurcation control, related work on stabiliza- 
tion, and applications of these techniques. The review papers [9,44] address control of chaos methods. 
In particular, [44] includes a discussion of use of sensitive dependence on initial conditions to direct tra- 
jectories to targets. The book [35] includes articles on control of chaos, detection of chaos in time series, 
chaotic data analysis, and potential applications of chaos in communication systems. The book [32] also 
contains discussions of control of bifurcations and chaos, and of analysis of chaotic data. 
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52.1 Introduction 


The interesting discovery that the topmost equilibrium of a pendulum can be stabilized by oscillatory 
vertical movement of the suspension point has been attributed to Bogolyubov [11] and Kapitsa [26], 
who published papers on this subject in 1950 and 1951, respectively. In the intervening years, literature 
appeared analyzing the dynamics of systems with oscillatory forcing, e.g., [31]. Control designs based 
on oscillatory inputs have been proposed (for instance [8] and [9]) for a number of applications. Many 
classical results on the stability of operating points for systems with oscillatory inputs depend on the 
eigenvalues of the averaged system lying in the left half-plane. Recently, there has been interest in the 
stabilization of systems to which such classical results do not apply. Coron [20], for instance, has shown 
the existence of a time-varying feedback stabilizer for systems whose averaged versions have eigenvalues 
on the imaginary axis. This design is interesting because it provides smooth feedback stabilization for 
systems which Brockett [15] had previously shown were never stabilizable by smooth, time-invariant 
feedback. For conservative mechanical systems with oscillatory control inputs, Baillieul [7] has shown 
that stability of operating points may be assessed in terms of an energy-like quantity known as the averaged 
potential. Control designs with objectives beyond stabilization have been studied in path-planning for 
mobile robots [40] and in other applications where the models result in “drift-free” controlled differential 
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equations. Work by Sussmann and Liu [36-38], extending earlier ideas of Haynes and Hermes, [21], 
has shown that, for drift-free systems satisfying a certain Lie algebra rank condition (LARC discussed in 
Section 52.2), arbitrary smooth trajectories may be interpolated to an arbitrary accuracy by appropriate 
choice of oscillatory controls. Leonard and Krishnaprasad [28] have reported algorithms for generating 
desired trajectories when certain “depth” conditions on the brackets of the defining vector fields are 
satisfied. 

This chapter summarizes the current theory of open-loop control using oscillatory forcing. The recent 
literature has emphasized geometric aspects of the methods, and our discussion in Sections 52.2 and 
52.3 will reflect this emphasis. Open-loop methods are quite appealing in applications in which the 
realtime sensor measurements needed for feedback designs are expensive or difficult to obtain. Because 
the methods work by virtue of the geometry of the motions in the systems, the observed effects may 
be quite robust. This is borne out by experiments described below. The organization of the article is 
as follows. In the present section we introduce oscillatory open-loop control laws in two very different 
ways. Example 52.1 illustrates the geometric mechanism through which oscillatory forcing produces 
nonlinear behavior in certain types of (drift-free) systems. Following this, the remainder of the section 
introduces a more classical analytical approach to control systems with oscillatory inputs. Section 52.2 
provides a detailed exposition of open loop design methods for so-called “drift-free” systems. The principal 
applications are in kinematic motion control, and the section concludes with an application to grasp 
mechanics. Section 52.3 discusses some geometric results of oscillatory forcing for stabilization. Examples 
have been chosen to illustrate different aspects of the theory. 


52.1.1 Noncommuting Vector Fields, Anholonomy, and the Effect of 
Oscillatory Inputs 


We begin by describing a fundamental mathematical mechanism for synthesizing motions in a controlled 
dynamical system using oscillatory forcing. We shall distinguish among three classes of systems: 


I. Drift-free systems with input entering linearly: 
m 
k=) > uigi(x). (52.1) 
i=l 


Here we assume each gj : R” > R” is a smooth (ie. analytic) vector field, and each “input” u;(-) is 
a piecewise analytic function of time. Generally, we assume m <n. 
II. Systems with drift and input entering affinely: 


k= f(x) + >> uigi(x). (52.2) 


i=1 


The assumptions here are, as in the previous case, with f : R” — R” also assumed to be smooth. 
III. Systems with no particular restriction on the way in which the control enters: 


x =f (x, u). (52.3) 


Here u = (u,...,Um)! is a vector of piecewise analytic inputs, as in the previous two cases, and 
f:R" x R™ > R" is analytic. 


This is a hierarchy of types, each a special case of its successor in the list. More general systems could be 
considered, and indeed, in the Lagrangian models which are described in Section 52.3, we shall encounter 
systems in which the derivatives of inputs also enter the equations of motion. It will be shown that these 
systems can be reduced to Class III, however. 
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Remark 52.1 


The extra term on the right hand side of Equation 52.2 is called a “drift” because, in the absence of control 
input, the differential equation “drifts” in the direction of the vector field f. 


Example 52.1: 


Even Class I systems possess the essential features of the general mechanism (anholonomy) by which 
oscillatory inputs may be used to synthesize desired motions robustly. (See remarks below on the 
robustness of open-loop methods.) Consider the simple and widely studied “Heisenberg” system 
(see [16]): 


xy uy(t) 
xyo)= u2(t) . (52.4) 
x3 ug (t)xy — uy (t)x2 


This system is a special case of Equation 52.1 in which m = 2 and 


1 0 
gix)=] O |, gaix)=]{ 1 
—X2 XI 


If we define the Lie bracket of these vector fields by [g1, g2] = oa 2Q- oe gi, then a simple calculation 
reveals [g1, 2] = (0,0, —2)? . Another fairly straightforward calculation shows that, from general con- 
siderations, there is a choice of inputs (u1(-), u2(-)) which generates a trajectory pointing approximately 
in the direction (0,0, 1)", and this approximation may be made arbitrarily accurate (see, Nijmeijer and 
Van der Schaft, [30], p. 77 or Bishop and Crittenden, [10], p. 18.). In the present case, we can be more 
precise and more explicit. Starting at the origin, (x1, x2,x3) = (0,0,0), motion in any of the three coor- 
dinate directions is possible. By choosing u;(t) = 1, u2(t) = 0, for instance, motion along the x; -axis is 
produced, and motion along the x-axis may similarly be produced by reversing the role of u;(-) and u2(-). 
Motion along the x3-axis is more subtle. If we let the inputs, (u1(-), u2(-)), trace a closed curve so that the 
states x; and x2 end with the same values with which they began, the net motion of the system is along the 
x3-axis. This is illustrated in Figure 52.1. Brockett [17] has observed that the precise shape of the input 


Anholonomy 


State space 
curve 


Input space curve 


<a 


FIGURE 52.1 The anholonomy present in the Heisenberg system is depicted in a typical situation in which the 
input variables trace a closed curve and the state variables trace a curve which does not close. The distance between 


endpoints of the state space curve (measured as the length of the dashed vertical line) reflects the anholonomy in the 
system. 
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curve is unimportant, but the x3-distance is twice the (signed) area circumscribed by the (x), x2)-curve. 
For this simple system, we thus have a way to prescribe trajectories between any two points in R*. Taking 
the case of trajectories starting at the origin, for instance, we may specify a trajectory passing through 
any other point (x,y,z) at time f = 1 by finding a circular arc initiating at the origin in (x1, x2)-space 
with appropriate length, initial direction, and curvature. This construction of inputs may be carried out 
somewhat more generally, as discussed below in Section 52.2. Ideas along this line are also treated in more 
depth in [28]. 

The geometric mechanism by which motion is produced by oscillatory forcing is fairly transparent in 
the case of the Heisenberg system. For systems of the form of Equation 52.2 and especially of the form 
of Equation 52.3, there is no comparably complete geometric theory. Indeed, much of the literature on 
such systems makes no mention of geometry. A brief survey/overview of some of the classical literature 
on such systems is given next. We shall return to the geometric point of view in Sections 52.2 and 52.3. 


52.1.2 Oscillatory Inputs to Control Physical Systems 


The idea of using oscillatory forcing to control physical processes is not new. Prompted by work in the 
1960s on the periodic optimal control of chemical processes, Speyer and Evans [33] derived a sufficiency 
condition for a periodic process to minimize a certain integral performance criterion. This approach 
also led to the observation that periodic paths could be used to improve aircraft fuel economy (see 
[32].). Previously cited work of Bogolyubov and Kapitsa [11] and [26], led Bellman et al. [8] and [9] 
to investigate the systematic use of vibrational control as an open loop control technique in which zero 
average oscillations are introduced into a system’s parameters to achieve a dynamic response (such as 
stabilizing effects). For example, it has been shown in [8,9,24,25], that the oscillation of flow rates in 
a continuous stirred tank reactor allows operating exothermic reactions at (average) yields which were 
previously unstable. Similarly, [6] and [7] describe the general geometric mechanism by which oscillations 
along the vertical support of an inverted pendulum stabilize the upper equilibrium point. 

This section treats the basic theory of vibrational control introduced in [8] and [9]. The techniques are 
primarily based on [8], with additional material taken from [24] and [25]. In particular, in [24] and [25], 
the technique of vibrational control has been extended to delay differential equations. 


52.1.3 Problem Statement 


Consider the nonlinear differential equation (Class III) 


dx 

2 D. 52.5 

di f(x, u) (52.5) 
where f : R” x R4 -> R” is continuously differentiable, x € R” is the state, and u=(w,..., ug)! isa 


vector of control inputs assumed to be piecewise analytic functions of time. These are the quantities 
which we can directly cause to vibrate. 

Introduce into Equation 52.5 oscillatory inputs according to the law u(t) = 9 + y(t) where Xo is a 
constant vector and y(t) is a periodic average zero (PAZ) vector. (For simplicity, y(t) has been assumed 
periodic. However, the following discussion can be extended to the case where y(t) is an almost periodic 
zero average vector [8,9,24,25].) Then Equation 52.5 becomes 


d. 
“ = f(x, do + y(t). (52.6) 


Assume that Equation 52.5 has a fixed equilibrium point x; = x.(o) for fixed u(t) = do. 
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Definition 52.1: 


An equilibrium point xs(o) of Equation 52.5 is said to be vibrationally stabilizable if, for any 8 > 0, there 
exists a PAZ vector y(t) such that Equation 52.6 has an asymptotically stable periodic solution, x*(t), 
characterized by 


1 rt 
|| x* —x,(Xo) ||< 8, where x* = =| x*(t) dt. 
0 


It is often preferable that Equations 52.5 and 52.6 have the same fixed equilibrium point, x;(Xo). 
However, this is not usually the case because the right hand side of Equation 52.6 is time varying and 
periodic. Therefore, the technique of vibrational stabilization is to determine vibrations y(f) so that the 
(possibly unstable) equilibrium point x;(Xo) bifurcates into a stable periodic solution whose average is 
close to x;(Xo). The engineering aspects of the problem consist of 1) finding conditions for the existence 
of stabilizing vibrations, 2) determining which oscillatory inputs, u(-), are physically realizable, and 3) 
determining the shape (waveform type, amplitude, phase) of the oscillations which will insure the desired 
response. In Section 52.3, we shall present an example showing how oscillatory forcing induces interesting 
stable motion in neighborhoods of points which are not close to equilibria of the time-varying system of 
Equation 52.6. 


52.1.4 Vibrational Stabilization 


It is frequently assumed that Equation 52.6 can be decomposed as 


d. 
_ = fi(x(t)) + flx(t), yd), (52.7) 


where Xo and y(-) are as above and where fi (x(t)) = fi(ho, x(t)) and the function f)(x(t), y(t) is linear 
with respect to its second argument. Systems for which this assumption does not hold are discussed in 
Section 52.3. For simplicity only, assume that f; and f) are analytic functions. Additionally, assume that 
y(t), the control, is periodic of period T(0 < T < 1) and in the form, y(t) = wii(wt), where w = x, 
and iu(-) is some fixed period-2x function (e.g., sin or cos). We write y(-) in this way because, although 
the theory is not heavily dependent on the exact shape of the waveform of the periodic input, there is a 
crucial dependence on the simultaneous scaling of the frequency and amplitude. Because we are usually 
interested in high frequency behavior, this usually implies that the amplitude of y(t) is large. It is possible, 
however, that i(-) has small amplitude, making the amplitude of y(t) small also. 
Under these assumptions, Equation 52.7 can be rewritten as 


d. 
— = fi(x(t)) + ofa(x(d), (wt). (52.8) 


To proceed with the stability analysis, Equation 52.8 will be transformed to an ordinary differential 
equation in “standard” form (2 = ¢f (x, t)) so that the method of averaging can be applied (see [11] 
and [24]). This allows the stability properties of the time varying system Equation 52.8 to be related to 
the stability properties of a simpler autonomous differential equation (the averaged equation). To make 
this transformation, consider the so-called “generating equation” 


dx 


a = fo(x(t), u(t). 


Suppose that this generating equation has a T-periodic general solution h(t, c), for some ii(-) and t > to, 
where h: R x R” > R” and c € R" is uniquely defined for every initial condition x(to) € Q C R". 
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Introduce into Equation 52.8 the Lyapunov substitution x(t) = h(wt,q(t)) to obtain an equation 
for q(-): 


dq | q(t)) 


-1 
dt aq fi(A(ot, q(t)))s 


which, in slow time t = wt, with z(t) = q(t) ande = i, becomes 


dz F | Oh[x, z(t)] 


-1 
Ae Dz filA(t, 2(1))). (52.9) 


Equation 52.9 is a periodic differential equation in “standard” form and averaging can be applied. If 
T denotes the period of the right hand side of Equation 52.9, then the averaged equation (autonomous) 
corresponding to Equation 52.9 is given as 


o = €Y (y(t); where 


zZ TT on 7 
Wo=7 i k =] fu(h(t,c)) dt. (52.10) 


By the theory of averaging, it is known that an €9 > 0 exists such that for 0 < € < €9, the hyperbolic 
stability properties of Equation 52.9 and Equation 52.10 are the same. Specifically, if y; is an asymptotically 
stable equilibrium point of Equation 52.10, this implies that, for 0 < € < €g, a unique periodic solution, 
z*(t) of Equation 52.9 exists, in the vicinity of y, that is asymptotically stable also. Since the transformation 
x(t) = h(wt, q(t)) is a homeomorphism, there will exist an asymptotically stable, periodic, solution to 
Equation 52.8 given by x* (t) = h(wt, z*(wt)) (converting back to fast time using the fact that q(t) = z(t), 
where z(-) is the solution to Equation 52.9). Using Definition 52.1, Equation 52.5 is said to be vibrationally 
stabilized provided that x* = t fe x*(t)dt remains in the vicinity of x;(o). This can be formalized by 
the following theorem given in [8] and [24]: 


Theorem 52.1: 


Assume that Equation 52.6 with y(t) = wit(wt) has the form of Equation 52.8, with f, and f, analytic. 
-1 

Assume, also, that h(t, c) is periodic and that the function [| fi(A(t, 2(t))) in Equation 52.8 is 

continuously differentiable with respect to z € 82 C R". Then the equilibrium point x.(do) of Equation 52.5 

is vibrationally stabilizable if a u(t) exists such that Equation 52.10 has an asymptotically stable equilibrium 

point characterized by t i. h(t, ys) dt = xs. 


The technique of vibrational control now becomes clearer. Introduce open loop oscillatory forcing into 
Equation 52.5, u(t) = 49 + wi(wt), such that Equation 52.5 is in the form of Equation 52.8. Transform 
Equation 52.8 into Equation 52.9 and study the stability properties of the corresponding average of 
Equation 52.9, given by Equation 52.10. Then determine parameters of i (phase, amplitude and frequency) 
such that Equation 52.10 has an asymptotically stable equilibrium point y,. If t i. h(t, ys) dt = xs, then 
the system is vibrationally stabilizable. 

The procedure of vibrational stabilization described above is trial and error. Vibrations are inserted 
into a system until vibrations are found which give the desired response. However, if specific classes of 
vibrations are analyzed, explicit algorithms are known which explain the size and location of vibration 
needed to give specified responses (see [8,9,24]). For example, it is common to assume that the vibrations 
are in linear multiplicative form, f,(x(t), y(t)) = B(t)x(t) or vector additive form, fx(x(t), y(t)) = L(t). In 
each of these cases, sufficient conditions (sometimes necessary and sufficient) are known for vibrational 
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FIGURE 52.2 A simple pendulum whose hinge point undergoes vertical motion. 


stabilization of systems. We return to linear multiplicative vibrations in Section 52.3, using a more 
geometric framework. 

Finally, it should be noted that [8] and [24] can relate the transient behavior of Equation 52.10 to the 
“average” transient behavior of Equation 52.7 through the estimation x(t) © h(wt, y(wt)), where y(t) is 
the solution of Equation 52.10 in slow time. Hence, the technique of vibrational control has the ability to 
both stabilize a system and control transient response issues. 


Example 52.2: Oscillatory Stabilization of a Simple Pendulum 


A classical example to which the theory applies involves stabilizing the upright equilibrium of a 
simple pendulum by the forced vertical oscillation of the pendulum’s hinge point. Consider a simple 
pendulum consisting of a massless but rigid link of length £ to which a tip of mass m is attached. 
Suppose the hinge point of the pendulum undergoes vertical motion, and is located at time t at 
vertical position R(t) with respect to some reference coordinate (see Figure 52.2.). Taking into account 
motion in this variable and the friction coefficient at the hinge (b > 0), the pendulum dynamics may 
be written as 


m b\. . 
0+ (=) 8+Rsin6+gsind =0. 
Consider the simple sinusoidal oscillation of the hinge-point, R(t) = asin Bt. Then R(t) = —mf sin Bt, 


where yn = (8) = af. Letting x; =68 and x27=8, the system can be placed in the form of 
Equation 52.8. The corresponding generating equation is given as x; = 0 and x2 = (n/@) sintsinx;, 
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which has solution x; =c, =hy(t,c) and x2 = —(n/£) cost sinc, + c¢2 = ho(t,c). Introducing the 
transformation x; =Z, and x2 =—(n/é)cosftsinz; +zZ2, letting t= ft, and letting «=1/f, 
Equation 52.9 specializes to 


Zz, =€ [- (7) cos tsinz; +22| 
n? g n 
Z=e€]— ra cos? tos Z} sinz, — (3) sinz, + (3) Z2 COS TCOSZ] 


fi nb ae b 
ay cos TSINZ1 — il Z2 |. 
Therefore the averaged equations are y;=«y2 and y2= e[ — a cosy siny; —(g/£) siny; 


— (b/mby2]. The upper equilibrium point in the averaged equation has been preserved 


and x; = a ie h(t, ys) dt. Hence, by the above theorem, if the vertical equilibrium point is asymp- 
totically stable for the averaged equation, then for sufficiently large B the inverted pendulum with 
oscillatory control has an asymptotically stable periodic orbit vibrating vertically about the point 
6= 1. A simple linearization of the averaged equation reveals that its upper equilibrium point is 
asymptotically stable when a2p2 > 2g. Under these conditions, the upper equilibrium point of the 
inverted pendulum is said to be vibrationally stabilized. 


52.2 The Constructive Controllability of Drift-Free 
(Class I) Systems 


Class I control systems arise naturally as kinematic models of mechanical systems. In this section, we 
outline the current theory of motion control for such systems, emphasizing the geometric mechanism 
(anholonomy) through which oscillatory inputs to Equation 52.1 produce motions of the state variables. 
Explicit results along the lines given in Section 52.1 for the Heisenberg system have been obtained in a 
variety of settings, and some recent work will be discussed below. The question of when such explicit 
constructions are possible more generally for Class I systems does not yet have a complete answer. Thus we 
shall also discuss computational approaches that yield useful approximate solutions. After briefly outlining 
the state of current theory, we conclude the section with an example involving motion specification for a 
ball “grasped” between two plates. 

The recent literature treating control problems for such systems suggests that it is useful distinguishing 
between two control design problems: 


Pl: The prescribed endpoint steering problem requires that, given any pair of points x9, x, € R”, a vector 
of piecewise analytic control inputs u(-) = (u1(-),...,Um(-)) is to be determined to steer the state 
of Equation 52.1 from xo at time t = 0 to x; at time t = T > 0. 

P2: The trajectory approximation steering problem requires that, given any sufficiently “regular” curve 
y : [0, T] — R", we determine a sequence {w(-)} of control input vectors such that the correspond- 
ing sequence of trajectories of Equation 52.1 converges (uniformly) to y. 


A general solution to either of these problems requires that a certain Lie algebra rank condition (LARC) 
be satisfied. More specifically, with the Lie bracket of vector fields defined as in Section 52.1, define a set 
of vector fields 


C={§ : §=[§),[&j-1,[--., [81,80]... Js 
& €(g1.....8m,i=lL....j fol...,c}. 


Then £ = span(C) (= the set of all linear combinations of elements of C) is called the Lie algebra 
generated by {gi,...,8m}. We say Equation 52.1 (or equivalently the set of vector fields {g1,...,m}) 
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satisfies the Lie algebra rank condition on R" if £ spans R" at each point of R”. The following result is 
fundamental because it characterizes those systems for which the steering problems may, in principle, be 
solved. 


Theorem 52.2: 


A drift-free system Equation 52.1 is completely controllable in the sense that, given any T > 0 and any pair 
of points xo, x, € R”, there is a vector of inputs u = (uy,...,Um) which are piecewise analytic on [0, T] and 
which steer the system from x to x; in T units of time if, and only if, the system satisfies the Lie algebra 
rank condition. 


As stated, this theorem is essentially due to W.L. Chow [19], but it has been refined and tailored to 
control theory by others ([13,35]). The various versions of this theorem in the literature have all been 
nonconstructive. Methods for the explicit determination of optimal (open-loop) control laws for steering 
Class I systems between prescribed endpoints have appeared in [1,2,16,18]. The common features in all 
this work are illustrated by the following: 

A model nonlinear optimal control problem with three states and two inputs: Find controls u(-), u2(-) 
which steer the system 


Xx) 0 0 u2 x) 
x2 = 0 0 Uy x2 (52. 1 1) 
x3 —u2 Uy, 0 x3 


between prescribed endpoints to minimize the cost criterion 


1 
n=[ uj + us dt. 
0 


Several comments regarding the geometry of this problem are in order. First, an appropriate version 
of Chow’s theorem shows that Equation 52.11 is controllable on any 2-sphere, S = {x € R?: ||x|| = 7} for 
some r > 0, centered at the origin in R*. Hence, the optimization problem is well-posed precisely when the 
prescribed endpoints x9, x; € R? satisfy ||xo|| = ||x1|]. Second, the problem may be interpreted physically 
as seeking minimum length paths on a sphere in which only motions composed of rotations about the 
x-axis (associated with input u,) and y-axis (associated with uz) are admissible. General methods for 
solving this type of problem appear in [1] and [2]. Specifically, in the first author’s 1975 PhD thesis 
(see reference in [1] and [2]), it was shown that the optimal inputs have the form, u(t) = pt sin(wt + 
), U2(t) = jr cos(wt + ¢). The optimal inputs depend on three parameters reflecting the fact that the set 
(group) of rotations of the 2-sphere is three dimensional. The details for determining the values of the 
parameters jt, w, and in terms of the end points x and x, are given in the thesis cited in [1] and [2]. 

The general nonlinear quadratic optimal control problem of steering Equation 52.1 to minimize a 
cost of the form fe; ||u||? dt has not yet been solved in such an explicit fashion. The general classes of 
problems which have been discussed in [1,2,16,18] are associated with certain details of structure in 
the set of vector fields {g1,..., 2m} and the corresponding Lie algebra £. In [16] and [18], for example, 
Brockett discusses various higher dimensional versions of the Lie algebraic structure characterizing the 
above sphere problem and the Heisenberg system. 

In addition to optimal control theory having intrinsic interest, it also points to a broader approach to 
synthesizing control inputs. Knowing the form of optimal trajectories, we may relax the requirement that 
inputs be optimal and address the simpler question of whether problems P1 and P2 may be solved using 
inputs with the given parametric dependence. Addressing the cases where we have noted the optimal 
inputs are phase-shifted sinusoids, we study the effect of varying each of the parameters. For instance, 
consider Equation 52.4 steered by the inputs u(t) = wt sin(wt + 9), u2(t) = p cos(wt + >), with p and > 
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fixed and w allowed to vary. As w increases, the trajectories produced become increasingly tight spirals 
ascending about the z-axis. One consequence of this is that, although the vectorfields 


1 
0 |, and 1 
x2 x1 


are both perpendicular to the x3-axis at the origin, pure motion in the x3-coordinate direction may 
nevertheless be produced to an arbitrarily high degree of approximation. This basic example can be 
generalized, and extensive work on the use of oscillatory inputs for approximating arbitrary motions in 
Class I systems has been reported by Sussmann and Liu, [36]. The general idea is that, when a curve 
y(t) is specified, a sequence of appropriate oscillatory inputs w/(-) is produced so that the corresponding 
trajectories of Equation 52.1 converge to y(-) uniformly. The interested reader is referred to [37,38], and 
the earlier work of Haynes and Hermes, [21], for further details. 

Progress on problem P1 has been less extensive. Although a general constructive procedure for gener- 
ating motions of Equation 52.1 which begin and end exactly at specified points, x9 and x1, has not yet been 
developed, solutions in a number of special cases have been reported. For the case of arbitrary “nilpo- 
tent” systems, Lafferriere and Sussmann, [22] and [23], provide techniques for approximating solutions 
for general systems. Leonard and Krishnaprasad [28] have designed algorithms for synthesizing open- 
loop sinusoidal control inputs for point-to-point system maneuvers where up to depth-two brackets are 
required to satisfy the LARC. 

Brockett and Dai [18] have studied a natural subclass of nilpotent systems within which the Heisenberg 
system Equation 52.4 is the simplest member. The underlying geometric mechanism through which a 
rich class of motions in R? is produced by oscillatory inputs to Equation 52.4 is also present in systems 
with two vector fields but higher dimensional state spaces. These systems are constructed in terms of 
nonintegrable p-forms in the coordinate variables x; and x2. We briefly describe the procedure, referring 
to [18] for more details. 

The number of linearly independent p-forms in x; and x2 is p+1. (Recall that a p-form in x, 
and x2 is a monomial of the form ako * The linearly independent p-forms may be listed explicitly 


{xt ; xP rious xb }.) Thus, there are 2(p + 1) linearly independent expressions of the form 
1 = (x1, X21 + W(21, X22 


where 6, are homogeneous polynomials of degree p in x; and x2. Within the set of such expressions, there 
is a set of p + 2 linearly independent expressions of the form n = dy/dt, where y is a homogeneous poly- 
nomial in x1, x2 of degree p + 1 (Such expressions are called exact differentials). There isa complementary 
p-dimensional family (p = 2(p + 1) — (p + 2)) of n’s which are not integrable. 

For example, if p = 2, there are 2 linearly independent nonintegrable forms ny, and we may take these 
to be {x{.2, x3}. From these, we construct a completely controllable two-input system 


xy u 

x2, Vv 

x3) =|) xv—xu]. (52.12) 
x4 xtv 

x5 xsu 


More generally, for each positive integer p we could write a completely controllable two-input system 
whose state space has dimension 2+ p(p+ 1)/2. Brockett and Dai [18] consider the optimal control 
problem of steering Equation 52.12 between prescribed endpoints x(0), x(T) € R° to minimize the cost 


functional ‘ 
i w+y dt. 
0 
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FIGURE 52.3 A ball rolling without slipping between two flat plates. 


It is shown that explicit solutions may also be obtained in this case, and these are given in terms of elliptic 
functions. 

Before treating an example problem in mechanics which makes use of these ideas, we summarize 
some other recent work on control synthesis for Class I systems. Whereas Sussmann and his coworkers 
[22,23,36-38] have used the concept ofa P. Hall basis for free Lie algebras to develop techniques applicable 
in complete generality to Class I systems, an approach somewhat in the opposite direction has been 
pursued by S. Sastry and his coworkers [29,39,40]. This approach sought to characterize systems controlled 
by sinusoidal inputs. Motivated by problems in the kinematics of wheeled vehicles and robot grasping, 
they have defined a class of chained systems in which desired motions result from inputs which are 
sinusoids with integrally related frequencies. While the results are no less explicit than the Heisenberg 
example in Section 52.1, the systems themselves are very special. Conditions under which a Class I system 
may be converted to chained form are given in [29]. 


Example 52.3: 


The example we discuss next (due to Brockett and Dai [18]) is prototypical of applications involving 
the kinematics of objects in the grasp of a robotic hand. Consider a ball that rolls without slipping 
between two flat horizontal plates. It is convenient to assume the bottom plate is fixed. Suppose that 
the ball has unit radius. Fix a coordinate system whose x- and y-axes lie in the fixed bottom plate with 
the positive z-axis perpendicular to the plate in the direction of the ball. Call this the (bottom) “plate 
frame.” We keep track of the ball’s motion by letting (x;,x2, 1) denote the plate-frame coordinates 
of the ball’s center. We also fix an orthonormal frame in the ball, and we denote the plate-frame 
directions of the three coordinate axes by a 3 x 3 orthogonal matrix 


a1 412. 443 
A= }421 422 433 


431 432 933 
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The ball’s position and orientation are thus specified by the 4 x 4 matrix 


411 412, 3X 

H#=(6 i= 421 422 423. -X2 
Oo 1 a31 432 433 ~CO#W? 

0 0 0 1 


As the top plate moves in the plate-frame x direction with velocity v1, the ball’s center also moves 
in the same direction with velocity u; = v; /2. This motion imparts a counterclockwise rotation about 
the y-axis, and since the ball has unit radius, the angular velocity is also uz. Similarly, if the top plate 
moves in the (plate-frame) y direction with velocity v2, the ball’s center moves in the y direction 
with velocity u2 = v2/2, and the angular velocity imparted about the x-axis is —uz. The kinematic 
description of this problem is obtained by differentiating H with respect to time. 


(2A i 
= (9) 


where 
0 0 Uy Uy 
Q={ 0 0 wW and u=|up2 
—Uuy —Uuj2 0 0 


This velocity relationship, at the point (x1, x2) = (0, 0), is 


H=u,U,;H+u2U>H, (52.13) 
where 
0 oO 1 1 0 oO O 0 
0 0 0 0 0 O 1 #1 
Y= 24-006 a |. Y= Lo et 00 
0 0 0 0 0 0 O 0 


Computing the Lie bracket of the vector fields U;H and U>H according to the formula given in 
Section 52.1, we obtain a new vector field U3H = [U,H, U2H], where 


= 


U3 = 


oCOo=-0 
coo! 

ocoo°o 
ooo°0 


Computing additional Lie brackets yields only quantities which may be expressed as linear com- 
binations of U;H, U2H, and U3H. Since the number of linearly independent vector fields obtained 
by taking Lie brackets is three, the system is completely controllable on a three dimensional space. 
Comparison with the problem of motion on a sphere discussed above shows that, by having the top 
plate execute a high-frequency oscillatory motion—uy (t) = w sin(wt + $), u2(t) = pp cos(wt + >), the 
ball may be made to rotate about its z-axis. The motion of the ball is “retrograde.” If the top plate 
executes a small amplitude clockwise loop about the “plate-frame” z-axis, the motion of the ball is 
counterclockwise. 


52.3 Systems with Drift—Stability Analysis Using 
Energy Methods and the Averaged Potential 


Constructive design methods for the control of systems of Class II are less well developed than in the 
Class I case. Nevertheless, the classical results on stability described in Section 52.2 apply in many cases, 
and the special structure of Class II systems again reveals the geometric mechanisms underlying stability. 
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52.3.1 First and Second Order Stabilizing Effects of Oscillatory Inputs 
in Systems with Drift 


Much of the published literature on systems of Class I and Class II is devoted to the case in which the 
vector fields are linear in the state. Consider the control system 


&= (A+) uj(t)B))x, (52.14) 
i=1 


where A, B),..., By, are constant m x n matrices, x(t) € R” with control inputs of the type we have been 
considering. We shall be interested in the possibility of using high-frequency oscillatory inputs to create 
stable motions in a neighborhood of the origin x = 0. 

Assume that i#(-),...,i%m(-) are periodic functions and (for simplicity only) assume that each i;(-) 
has common fundamental period T > 0. To apply classical averaging theory to study the motion of 
Equation 52.14, we consider the effect of increasing the frequency of the forcing. Specifically, we study 
the dynamics of 


X(t) = (A+) > ai(wot)Bi)x(t) 


as w becomes large. The analysis proceeds by scaling time and considering t = wt. Let z(t) = x(t). This 


satisfies the differential equation 


dz 1 A 
z= S(A+ D) ti(e)Bidz. (52.15) 


Assuming w > 0 is large, and letting « = i, we see that Equation 52.15 is in a form to which classical 
averaging theory applies. 


Theorem 52.3: 


Consider the system of Equation 52.14 with u;(-) = u,(-) where, for i=1,...,m, uj(-) is continuous on 
0 <t < ty < co and periodic of period T « ty. Let 


1 T 
“=z i in;(t) dt, 
T Jo 
and let y(t) be a solution of the constant coefficient linear system 


H(t) = (A+ > UBi)y(x). (52.16) 


If zp and yo are respective initial conditions associated with Equations 52.15 and 52.16 such that |zo — yo| = 
Ole), then |z(t) — y(t)| = Ole) on a time scale t ~ I If A+). u,B; has its eigenvalues in the left half 
plane, then |z(t) — y(t)| = Ole) as t—> 00. 


This theorem relies on classical averaging theory and is discussed in [7]. A surprising feature of 
systems of this form is that the stability characteristics can be modified differently if both the magnitude 
and frequency of the oscillatory forcing are increased. A contrast to Theorem 52.3 is the following: 


Theorem 52.4: 


Let ity (-),..., Um(-) be periodic functions of period T > 0 fori=1,...,m. Assume that each uj(-) has mean 
0. Consider Equation 52.14, and assume that, for alli,j=1,...,m,B;Bj =0. Let e = 1. Define for each 


oO 
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i=1,...,m, the periodic function v;(t) = 4, i;(s) ds, and let 


1 rt 
=z vi(s)ds, i=1,...,m, 
T Jo 


and 


1 rt 
w= 7 f vi(s)vj(s) ds, i,j=1,...,m. 


Let y(t) be a solution of the constant coefficient linear system 


J =| A+) (9) — 04) BAB; | y. (52.17) 
ij 


Suppose that the eigenvalues of Equation 52.17 have negative real parts. Then there is a t; > 0 such that 
for w > 0 sufficiently large (i.e., for € > 0 sufficiently small), if xo and yo are respective initial conditions for 
Equations 52.14 and 52.17 such that |xo — yo| = O(e), then |x(t) — y(t)| = O(€) for allt > ty. 


The key distinction between these two theorems is that induced stability is a first order effect in 
Theorem 52.3 where we scale frequency alone, whereas in Theorem 52.4 where both frequency and 
magnitude are large, any induced stability is a second order effect (depending on the rms value of the 
integral of the forcing). Further details are provided in [7]. Rather than pursue these results, we describe 
a closely related theory which may be applied to mechanical and other physical systems. 


52.3.2 A Stability Theory for Lagrangian and Hamiltonian Control Systems 
with Oscillatory Inputs 


The geometric nature of emergent behavior in systems subject to oscillatory forcing is apparent in the 
case of conservative physical systems. In this subsection, we again discuss stabilizing effects of oscillatory 
forcing. The main analytical tool will be an energy-like quantity called the averaged potential. This is 
naturally defined in terms of certain types of Hamiltonian control systems of the type studied in [30]. 
Because we shall be principally interested in systems with symmetries most easily described by Lagrangian 
models, we shall start from a Lagrangian viewpoint and pass to the Hamiltonian description via the 
Legendre transformation. Following Brockett, [14] and Nijmeijer and Van der Schaft, [30], we define 
a Lagrangian control system on a differentiable manifold M as a dynamical system with inputs whose 
equations of motion are prescribed by applying the Euler-Lagrange operator to a function L: TM x U > 
R, L = L(q, q; u), whose dependence on the configuration q, the velocity g, and the control input u is 
smooth. U isa set of control inputs satisfying the general properties outlined in Section 52.1. 

Lagrangian systems arising via reduction with respect to cyclic coordinates: Consider a Lagrangian 
control system with configuration variables (q1, q2) € R”! x R". The variable q; will be called cyclic if it 
does not enter into the Lagrangian when u = 0, ice., if ba (413 92> 41» G2; 0) = 0. A symmetry is associated 
with the cyclic variables q; which manifests itself in the invariance of L(q1, q2; 41, 42; 0) with respect to a 
change of coordinates q; +> q1 + a for any constant a € R”. We shall be interested in Lagrangian control 
systems with cyclic variables in which the cyclic variables may be directly controlled. In such systems, we 
shall show how the velocities associated with the cyclic coordinates may themselves be viewed as controls. 
Specifically, we shall consider systems of the form 


L(qi, 425 41 423 u) = L( qr, Gi, 42) +4) u (52.18) 


where 


~ ey Lie ery (mq) ATQ)\ (im 
£42, Gi 42) = 5 (41 > 42) ee 72) e) — V(qp). 
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The matrices m(qz2) and M(q2) are symmetric and positive definite of dimension , x n, and nz x nz 
respectively, where dim q; = nj, dim q2 = no, and the matrix A(qz) is arbitrary. 

To emphasize the distinguished role to be played by the velocity associated with the cyclic variable q1, 
we write v = q,. Applying the usual Euler-Lagrange operator to this function leads to the equations of 
motion, 


and 


(5 oF 5) =, (52.19) 
042 lc 


92+42>¥) 


The first of these equations may be written more explicitly as 


d 
a linlaadv + Alga)! 4p) Sh (52.20) 


Although, in the physical problem represented by the Lagrangian Equation 52.18, u(-) is clearly the 
input with v(-) determined via Equation 52.20, it is formally equivalent to take v(-) as the input with 
the corresponding u(-) determined by Equation 52.20. In actual practice, this may be done, provided we 
may command actuator inputs u(-) large enough to dominate the dynamics. The motion of q is then 
determined by Equation 52.19 with the influence of the actual control input felt only through v(-) and v(-). 
Viewing v(-) together with v(-) as control input, Equation 52.19 is a Lagrangian control system in its own 
right. The defining Lagrangian is given by L(q, 423 v) = 543 M(q2)q2 + q3 A(qo)v — Va(q2; v), where Vg 
is the augmented potential defined by Va(q; v) = V(q) — $v! m(q)v. In the remainder of our discussion, 
we shall confine our attention to controlled dynamical systems arising from such a reduced Lagrangian. 
Because the reduction process itself will typically not be central, we henceforth omit the subscript “2” on 
the generalized configuration and velocity variables which we wish to control. We write: 


L(q, qv) = si M@a+ a Ay — Va(qs v). (52.21) 


Example 52.4: The Rotating Pendulum 


As in [4], we consider a mechanism consisting of a solid uniform rectangular bar fixed at one end to 
a universal joint as depicted in Figure 52.4. The universal joint is comprised of two single degree of 
freedom revolute joints with mutually orthogonal intersecting axes (labeled x and y in Figure 52.2). 
These joints are assumed to be frictionless. Angular displacements about the x- and y-axes are 
denoted $1 and 2 respectively, with (1,2) = (0,0) designating the configuration in which the 
pendulum hangs straight down. The pendulum also admits a controlled rotation about a spatially 
fixed vertical axis. Let 6 denote the amount of this rotation relative to some chosen reference 
configuration. 


To describe the dynamics of the forced pendulum, we choose a (principal axis) coordinate frame, fixed 
in the bar, consisting of the x- and y-axes of the universal joint together with the corresponding z-axis 
prescribed by the usual right-hand rule. When the pendulum is at rest, for some reference value of 0, 
the body frame x—,y—, and z-axes will coincide with corresponding axes x’, y’, and z’ of an inertial 
frame, as in Figure 52.4, with respect to which we shall measure all motion. Let I,,, Jy, and I, denote the 
principal moments of inertia with respect to the body (x, y,z)-coordinate system. Then the system has 
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FIGURE 52.4 A rotating pendulum suspended from a universal joint. 
the Lagrangian 


L(1, 15 2, 6256) = = [Le(Os2 + $1)? + Ly (Ocas1 — 201)" + Ie(Oerc2 + $281)"] + e102, (52.22) 


1 
2 
where the last term is a normalized gravitational potential, and where s; = sin 4;, cj = cos b;. We assume 
that there is an actuator capable of rotating the mechanism about the inertial z-axis with any prescribed 
angular velocity 0. We shall study the dynamics of this system when I, > I, > Iz, and as above, we shall 
view v = 0(-) as a control input. The reduced Lagrangian Equation 52.21 takes the form 


L(, 3 v) = = [Iebt + (yey + Les7) 63] + v [Iesob1 + (Lz — Iy)sicicrda] 


1 
2 
+ [(lecy + Iyst)cy + Les3] v? + e102. (52.23) 
The corresponding control system is given by applying the Euler-Lagrange operator to L, and this is 
represented by a system of coupled second order differential equations, 


a a 


aT a 0. (52.24) 


From the way in which v appears in the reduced Lagrangian, the equations of motion Equation 52.24 will 
have terms involving ¥ as well as terms involving v. Though it is possible to analyze such a system directly, 
we shall not discuss this approach. The general analytical technique advocated here is to transform all 
Lagrangian models to Hamiltonian form. 

The Hamiltonian viewpoint and the averaged potential: Recall that in Hamiltonian form the dynamics 


are represented in terms of the configuration variables q and conjugate momenta p = at (see [30], p. 351). 
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Referring to the Lagrangian in Equation 52.21 and applying the Legendre transformation H(q, p;v) = 
[p-q—L(q, 4 V)]i(qp)» We obtain the corresponding Hamiltonian 


A(q, ps v) = 50 — vA)’ M~!(p—vA)+ Va(v,q). (52.25) 


The equations of motion are then written in the usual way in position and momentum coordinates as 


qg=M '(p—vA) (52.26) 
Ce Lee vary cree 
p= Og EG vA)’ M~"(p— vA) + Va(v, a| : (52.27) 


We may obtain an averaged version of this system by replacing all coefficients involving the input v(-) by 
their time-averages. Assuming v(-) is bounded, piecewise continuous, and periodic of period T > 0, v(-) 
has a Fourier series representation: 


lo.) 
wk + 
v(t)= >> et". (52.28) 
k=—00 


Equations 52.26 and 52.27 contain terms of order not greater than two in v. Averaging the coefficients we 
obtain 


Proposition 52.1: 


Suppose v(-) is given by Equation 52.28. Then, if all coefficients in Equations 52.26 and 52.27 are replaced 
by their time averages, the resulting averaged system is Hamiltonian with corresponding Hamiltonian 
function 


H(qp)= 5 — A(g))"M(q)"(p — A(q)¥) + Va), (52.29) 
where 
Va(q) = V(q)4 5 (2@) *A(g) Mg)" 'A(Q)*) , (52.30) 
1 T 
y= T i Vv dt, 
0 
and 
1 T 
LQ) =F : v(t)? (A@™M@"'A(@) - m(q)) v(t) dt. 
0 
Definition 52.2: 


We refer to H(q, p) in Equation 52.29 as the averaged Hamiltonian associated with Equation 52.25. Va(q), 
defined in Equation 52.30, is called the averaged potential. 


Remark 52.2 


(Averaged kinetic and potential energies, the averaged Lagrangian.) Before describing the way in which 
the averaged potential may be used for stability analysis, we discuss its formal definition in more detail. 
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The Legendre transformation used to find the Hamiltonian corresponding to Equation 52.21 makes use 
of the conjugate momentum 


ee)! 
P=P(94t)= 3a = M(q)q+ A(q)v(t). 
This explicitly depends on the input v(t). Given a point in the phase space, (q,q), the corresponding 
averaged momentum is 
p=M(q)q+ AQ. 


We may think of the first term 5(p — A(q)¥)" M(q)\(p — A(q)v) in Equation 52.29 as an “averaged 
kinetic energy.” It is not difficult to see that there is an “averaged Lagrangian” 


1q.d) = 54" M@i +4" A@?— Vala) 


from which the Hamiltonian H in Equation 52.29 is obtained by means of the Legendre transformation. 

The averaged potential is useful in assessing the stability of motion in Hamiltonian (or Lagrangian) 
control systems. The idea behind this is that strict local minima of the averaged potential will correspond 
to stable equilibria of the averaged Hamiltonian system. The theory describing the relationship with the 
stability of the forced system is discussed in [6] and [7]. The connection with classical averaging is empha- 
sized in [6], where Rayleigh dissipation is introduced to make the critical points hyperbolically stable. 
In [7], dissipation is not introduced, and a purely geometric analysis is applied within the Hamiltonian 
framework. We state the principal stability result. 


Theorem 52.5: 


Consider a Lagrangian control system prescribed by Equation 52.21 or its Hamiltonian equivalent (Equation 
52.25). Suppose that the corresponding system of Equations 52.26 and 52.27 is forced by the oscillatory input 
given in Equation 52.28. Let qo be a critical point of the averaged potential which is independent of the period 
T (or frequency) of the forcing. Suppose, moreover, that, for all T sufficiently small (frequencies sufficiently 
large), qo is a strict local minimum of the averaged potential. Then (q, q) = (qo, 0) is a stable equilibrium 
of the forced Lagrangian system, provided T is sufficiently small. If (q, p) = (qo,0) is the corresponding 
equilibrium of the forced Hamiltonian system, then it is likewise stable, provided T is sufficiently small. 


This theorem is proved in [7]. 
We end this section with two examples to which this theorem applies, followed by a simple example 
which does not satisfy the hypothesis and for which the theory is currently less complete. 


Example 52.5: Oscillatory Stabilization of a Simple Pendulum 


Consider, once again, the inverted pendulum discussed in Example 52.2. Assume now that no 
friction exists in the hinge and, therefore, b = 0. Using the classical theory of vibrational control in 
Example 52.6, it is not possible to draw conclusions on the stabilization of the upper equilibrium point 
by fast oscillating control, because the averaged equation will have purely imaginary eigenvalues 
when b = 0. The averaged potential provides a useful alternative in this case. For b = 0, the pendulum 
dynamics may be written 

06+Rsin6+gsin0 = 0, 


where all the parameters have been previously defined in Example 52.2. Writing the pendulum’s ver- 
tical velocity as v(t) = R(t), this is a system of the type we are considering with (reduced) Lagrangian 


Open-Loop Control Using Oscillatory Inputs 52-19 


Le, 6;v) = (1 /2)662 +v0sin 6+ gcos 9. To find stable motions using the theory we have presented, 
we construct the averaged potential by passing to the Hamiltonian description of the system. The 


momentum (conjugate to 8) is p= & = (6+vsin8. Applying the Legendre transformation, we 


obtain the corresponding Hamiltonian 


—(p—vsin0)* —gcos90. 


H(@, Div) = (28 — Diag = 5 


If we replace the coefficients involving v(-) with their time-averages over one period, we obtain the 
averaged Hamiltonian 


= 1 1 
H(6, p) = —(p —vsin 6)? + —(= — Vv) sin? 6 — gcos 6, 
(8, p) rT) (p )* + 5 7! ) 9g 
where v and © are the time averages over one period of v(t) and v(t)2 respectively. The averaged 
potential is just V4(6) = H(z — v2) sin2 9 — gcos 9. Consider the simple sinusoidal oscillation of the 
hinge-point, R(t) = asin Bt. Then v(t) = af cos Bt. Carrying out the construction we have outlined, 
the averaged potential is given more explicitly by 


2a2 
TAC ee 


sin? 6 — gcos 0. (52.31) 


Looking at the first derivative V4 (9), we find that 0 = x is a critical point for all values of the param- 
eters. Looking at the second derivative, we find that Va (rt) > 0 precisely when 2B > 2g. From 
Theorem 52.5 we conclude that, for sufficiently large values of the frequency £, the upright equilib- 
rium is stable in the sense that motions of the forced system will remain nearby. This is of course 
completely consistent with classical results on this problem. (cf. Example 52.2.) 


Example 52.6: 


Example 52.4, reprise: oscillatory stabilization of a rotating pendulum. 


Let us return to the mechanical system treated in Example 52.4. Omitting a few details, we proceed 
as follows. Starting from the Lagrangian in Equation 52.23, we obtain the corresponding Hamiltonian 
(the general formula for which is given by Equation 52.25). The averaged potential is given by the 
formula in Equation 52.30. Suppose the pendulum is forced to rotate at a constant rate, perturbed 
by a small-amplitude sinusoid, v(t) = w + asin Bt. Then the coefficients in Equation 52.30 are 


B 2n/B 

v= 5a v(t) dt = w, and 
Tw JO 
B 2n/B 5 5 a2 


and some algebraic manipulation shows that the averaged potential is given in this case by 


1 wees 
4 lyct + Izs5 2 


[53 + (lys4 + l2ch)c3 | wo’. 


Valor, 62) = —€1¢2 


Stable modes of behavior under this type of forcing correspond to local minima of the averaged 
potential. A brief discussion of how this analysis proceeds will illustrate the utility of the approach. 


When a = 0, the pendulum undergoes rotation at a constant rate about the vertical axis. For all rates w, 
the pendulum is in equilibrium when it hangs straight down. There is a critical value, w,,, however, above 
which the vertical configuration is no longer stable. A critical point analysis of the averaged potential 
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yields the relevant information and more. The partial derivatives of V4 with respect to ; and ¢2 both 
vanish at (1,2) = (0,0) for all values of the parameters a, B, w. To assess the stability of this critical 
point using Theorem 52.5, we compute the Hessian (matrix of second partial derivatives) of V4 evaluated 


at (1, b2) = (0, 0): 


V4 VA II 
7 00) 2 {-(0,0) Z 2 2 
a0? 36106 1+ 57 (I, — Iya? — (Iy — Iz) 0 
ov. Orv. . 1 
4 (0,0) —(o,0) 0 1+ <1,0? — (Ik — I,)w? 
0o1 92 005 2 


Let us treat the constant rotation case first. We have assumed I, > >. When a = 0, this means that 
the Hessian matrix above is positive definite for 0 < w” < 1/(I, —I,). This inequality gives the value of 
the critical rotation rate precisely as wr = 1/./I; — Iz. We wish to answer the following question: Is it 
possible to provide a stabilizing effect by superimposing a small-amplitude, high-frequency sinusoidal 
oscillation on the constant-rate forced rotation? The answer emerges from Theorem 52.5 together with 
analysis of the Hessian. In the symmetric case, I, = Iy, the answer is “no” because any nonzero value of 
a will decrease the (1, 1)-entry and hence the value of w,;. If I, > Iy, however, there is the possibility of 
increasing w,, slightly, because, although the (1, 1)-entry is decreased, the more important (2, 2)-entry is 
increased. 

Current research on oscillatory forcing to stabilize rotating systems (chains, shafts, turbines, etc.) is 
quite encouraging. Though only modest stabilization was possible for the rotating pendulum in the 
example above, more pronounced effects are generally possible with axial forcing. Because this approach 
to control appears to be quite robust (as seen in the next example), it merits attention in applications 
where feedback designs would be difficult to implement. 

We conclude with an example to which Theorem 52.5 does not apply and for which the theory is 
currently less well developed. Methods of [6] can be used in this case. 


Example 52.7: 


Oscillation induced rest points in a pendulum ona cart. We consider a slight variation on Example 52.5 


wherein we consider oscillating the hinge point of the pendulum along a line which is not vertical. 
More specifically, consider a cart to which there is a simple pendulum (as described in Example 52.5) 
attached so that the cart moves along a track inclined at an angle W to the horizontal. Suppose 
the position of the cart along its track at time t is prescribed by a variable r(t). Then the pendulum 
dynamics are expressed 

£0 +¥cos(@— y)+gsind = 0. 


Note that when = 1/2, the track is aligned vertically, and we recover the problem treated in 
Example 52.5. In the general case, let v(t) = 7(t) and write the averaged potential 


1 
Va(0) = —gcos0+ =, cos?(6 — a) (Z— v7?) 


where 


and 
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As in Example 52.5, we may take v(t) = af cos ®t. For sufficiently large frequencies 8 there are 
strict local minima of the averaged potential which are not equilibrium points of the forced system. 
Nevertheless, as noted in [6], the pendulum will execute motions in a neighborhood of such a point. 
To distinguish such emergent behavior from stable motions in neighborhoods of equilibria (of the 
nonautonomous system), we have called motions confined to neighborhoods of nonequilibrium 
critical points of the averaged potential hovering motions. For more information on such motions, 
the reader is referred to [41]. 


Remark on the Robustness of Open-Loop Methods 


The last example suggests, and laboratory experiments bear out, that the stabilizing effects of oscillatory 
forcing of the type we have discussed are quite pronounced. Moreover, they are quite insensitive to the fine 
details of the mathematical models and to physical disturbances which may occur. Thus, the stabilizing 
effect observed in the inverted pendulum will be entirely preserved if the pendulum is perturbed or if the 
direction of the forcing isn’t really vertical. Such robustness suggests that methods of this type are worth 
exploring in a wider variety of applications. 


Remark on Oscillatory Control with Feedback 


There are interesting applications (e.g., laser cooling) where useful designs arise through a combination 
of oscillatory forcing and certain types of feedback. For the theory of time-varying feedback designs, 
the reader is referred to [20] and the chapters on stability by Khalil, Teel, Sontag, Praly, and Georgiou 
appearing in this handbook. 


52.4 Defining Terms 


Anholonomy: Consider the controlled differential equation 52.1, and suppose that there is a non-zero 
function @ : R” x R” — R such that (x, gi(x)) = 0 for i= 1,...,m. This represents a constraint 
on the state velocities which can be commanded. Despite such a constraint, it may happen that any 
two specified states can be joined by a trajectory x(t) generated via Equation 52.1 by an appropriate 
choice of inputs u;(-). Any state trajectory arising from Equation 52.1 constrained in this way is said 
to be determined from the inputs u;(-) by anholonomy. In principle, the notation of anholonomy 
can be extended to systems given by Equation 52.2 or 52.3. Some authors who were consulted in 
preparation of this chapter objected to the use of the word in this more general context. 

Averaged potential: An energy-like function that describes the steady-state behavior produced by high- 
frequency forcing of a physical system. 

Completely controllable: A system of Equation 52.3 is said to be completely controllable if, given any 
T > 0 and any pair of points xo, x; € R”, there is a control input u(-) producing a motion x(-) of 
Equation 52.3 such that x(0) = xo and x(T) = x). 

LARC: The Lie algebra rank condition is the condition that the defining vector fields in systems, such as 
Equation 52.1 or 52.2 together with their Lie brackets of all orders span R”. 
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53.1 Introduction: Backstepping 


Realistic models of physical systems are nonlinear and usually contain parameters (masses, inductances, 
aerodynamic coefficients, etc.) which are either poorly known or dependent on a slowly changing 
environment. If the parameters vary in a broad range, it is common to employ adaptation: a parameter 
estimator—identifier—continuously acquires knowledge about the plant and uses it to tune the controller 
“on-line.” 

Instabilities in nonlinear systems can be more explosive than in linear systems. During the parameter 
estimation transients, the state can “escape” to infinity in finite time. For this reason, adaptive nonlinear 
controllers cannot simply be the “adaptive versions” of standard nonlinear controllers. 

Currently, the most systematic methodology for adaptive nonlinear control design is backstepping. We 
introduce the idea of backstepping by carrying out a nonadaptive design for the system 


x, =x2 +(x)", (0) =0 (53.1) 
fo =u, (53.2) 


where 0 is a known parameter vector and ¢(x,) is a smooth nonlinear function. Our goal is to stabilize 
the equilibrium x; = 0, x. = —¢(0)'0 = 0. Backstepping design is recursive. First, the state x is treated 
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as a virtual control for the x-equation 53.1, and a stabilizing function 
ay (x1) = —c1x1 — @(x1)"@, ce) > 0 (53.3) 


is designed to stabilize (Equation 53.1) assuming that x2 = a 1(x,) can be implemented. Since this is not 
the case, we define 


Zz =X, (53.4) 
Z2 = x2 — a1 (x1), (53.5) 


where 2) is an error variable expressing the fact that x2 is not the true control. Differentiating z; and zz 
with respect to time, the complete system (Equations 53.1 and 53.2) is expressed in the error coordinates 
(Equations 53.4 and 53.5): 


Z =X, = x24 p0=nm+a,+9!0= —C1Z| +22, (53.6) 

: ; ; Oy . Oa 

2=%2-A =u 4 =u : (x2 + 9°6) : (53.7) 
Oxy Ox 


It is important to observe that the time derivative 4 is implemented analytically, without a differen- 
tiator. For the system (Equations 53.6 and 53.7), we now design a control law u = a2(x1, x2) to render the 
time derivative of a Lyapunov function negative definite. It turns out that the design can be completed 
with the simplest Lyapunov function 


1 1 
V(x1, x2) = -27 + 23. (53.8) 
2 2 
Its derivative for Equations 53.6 and 53.7 is 
F Ou 
Vay (-cazmta)+z E ape (x + o*e) | 
Oxy 
2 Oay T 
= cz, +22 )utz— a (a+¢ 8) ‘ (53.9) 
1 


An obvious way to achieve negativity of V is to employ u to make the bracketed expression equal to 
—c2Z2 with cz > 0, namely, 


0 
v= ose es) Soe 215 (x. +97). (53.10) 
x) 


This control may not be the best choice because it cancels some terms which may contribute to the 
negativity of V. Backstepping design offers enough flexibility to avoid cancellation. However, for the sake 
of clarity, we assume that none of the nonlinearities is useful, so that they all need to be cancelled as in 
the control law (Equation 53.10). This control law yields 


V= —cz7 _ 0225, (53.11) 


which means that the equilibrium z = 0 is globally asymptotically stable. In view of Equations 53.4 and 
53.5, the same is true about x = 0. The resulting closed-loop system in the z-coordinates is linear: 


els : Wey (53.12) 
22 —-1l —c, 22 


In the next four sections we present adaptive nonlinear designs through examples. Summaries of 
general design procedures are also provided but without technical details, for which the reader is referred 
to the text on nonlinear and adaptive control design by Krsti¢ et al. [9]. Only elementary background on 
Lyapunov stability is assumed, while no previous familiarity with adaptive linear control is necessary. The 
two main methodologies for adaptive backstepping design are the tuning functions design, Section 53.2, 
and the modular design, Section 53.3. These sections assume that the full state is available for feedback. 
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Section 53.4 presents designs where only the output is measured. Section 53.5 discusses various extensions 
to more general classes of systems, followed by a brief literature review in Section 53.6. 


53.2 Tuning Functions Design 


In the tuning functions design, both the controller and the parameter update law are designed recursively. 
At each consecutive step a tuning function is designed as a potential update law. The tuning functions are 
not implemented as update laws. Instead, the stabilizing functions use them to compensate the effects of 
parameter estimation transients. Only the final tuning function is used as the parameter update law. 


53.2.1 Introductory Examples 


The tuning functions design will be introduced through examples with increasing complexity: 


A B Cc 
xy = Ut 9(x1)10 dy =x + (x1)7O = x2 + G(x1)"O 
x2 =u x2 = X3 
x3 =u 


The adaptive problem arises because the parameter vector 0 is unknown. The nonlinearity (x) is known 
and for simplicity it is assumed that ~(0) = 0. The systems A, B, and C differ structurally: the number 
of integrators between the control u and the unknown parameter @ increases from zero at A, to two at 
C. Design A will be the simplest because the control uv and the uncertainty @(x1)'@ are “matched,” that 
is, the control does not have to overcome integrator transients in order to counteract the effects of the 
uncertainty. Design C will be the hardest because the control must act through two integrators before it 
reaches the uncertainty. 


53.2.1.1 Design A 
Let 6 be an estimate of the unknown parameter 0 in the system 
x =ut+q'0. (53.13) 
If this estimate were correct, 6 = 0, then the control law 
u=—c) x) — (x1) "6 (53.14) 


would achieve global asymptotic stability of x = 0. Because 6 = 0 — 6 £ 0, we have 


xy = —c, x + (x1) 8, (53.15) 


that is, the parameter estimation error 6 continues to act as a disturbance which may destabilize the 
system. Our task is to find an update law for 6(t) which preserves the boundedness of x(t) and achieves 
its regulation to zero. To this end, we consider the Lyapunov function 


Vi (x, 6) = = a3 sor, (53.16) 
where I’ is a positive definite symmetric matrix. The derivative of Vj is 

Vi = cx? + x1 0(x;)"6— §'r-16 = cx? ert (Foca )x1 — 8) : (53.17) 
Our goal is to select an update law for 6 to guarantee 


Vi <0. (53.18) 
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The only way this can be achieved for any unknown 0 is to choose 
6 =Te(x1)x1. (53.19) 


This choice yields 
Vi = —c1x?, (53.20) 


which guarantees global stability of the equilibrium x; = 0,6 =6, and hence, the boundedness of x; (t) 
and 0(t). By LaSalle’s invariance theorem (see Chapter 39 by Khalil in this book), all the trajectories of the 
closed-loop adaptive system converge to the set where V; = 0, that is, to the set where c1x} = 0, which 
implies that 
lim x,(t)=0. (53.21) 
too 


Alternatively, we can prove Equation 53.21 as follows. By integrating Equation 53.20 we obtain 
ve 1x1 (t)?2dt = Vi (x1 (0), 8(0)) — Vi (x1 (t), 6(t)), which, thanks to the nonnegativity of V;, implies that 
if 1x1 (t)2dt < Vi (x; (0), 6(0)) < oo. Hence, x; is square-integrable. Due to the boundedness of x; (t) and 


A(t), from Equations 53.15 and 53.19 it follows that x; (t) and 6 are also bounded. By Barbalat’s lemma we 
conclude that x(t) > 0. 

The update law (Equation 53.19) is driven by the vector (x1), called the regressor, and the state x). 
This is a typical form of an update law in the tuning functions design: the speed of adaptation is dictated 
by the nonlinearity p(x,) and the state x). 


53.2.1.2 Design B 


For the system 


iy = x2 + (x1), 
See (53.22) 


we have already designed a nonadaptive controller in Section 53.1. To design an adaptive controller, we 
replace the unknown 9 by its estimate both in the stabilizing function (Equation 53.3) and in the change 
of coordinate (Equation 53.5): 

2 =x —041(x1,9), a4 (1,6) = —cz1 — 976. (53.23) 
Because in the system (Equation 53.22) the control input is separated from the unknown parameter by 
an integrator, the control law (Equation 53.10) will be strengthened by a term v2(x1,x2,9) which will 
compensate for the parameter estimation transients: 


A Oa rn PR 
U = 07(x],%2,9) = —Oz2 —zy + a (x + g"4) + v2(x1, x2, 9). (53.24) 
1 


The resulting system in the z coordinates is 


Z=a+t+a,+9)0=-cz, ++ 916, (53.25) 
Oa Oa * 
22 =%X2-A) =U (x + 9") - +6 
Ox 
Oa ~ Ody4 - 
= —Z1 — (222 ~ 976 ~ 6+ vo(x1, x2; 6), (53.26) 
Ox) 08 
or, in vector form, 
g: 0 


al —C] 1 [2 ~ 
oh Ih Ae 6+ : 2 lis (53.27) 
] | -1 -c 22 ais — 8+ vale a8) 
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The term v2 can now be chosen to eliminate the last brackets: 


a Oa, * 
Vv2(x1, x2, 8) = aay (53.28) 
00 


This expression is implementable because 6 will be available from the update law. Thus we obtain the 


error system 
z c 1 Zz : 
1 Ce) 1 A 
_ |= 0. 3.2 
[2 al, SO B “i oe gr a 


When the parameter error 4 is zero, this system becomes the linear asymptotically stable system (Equation 


53.12). Our remaining task is to select the update law 0 = I't2(x, 0). Consider the Lyapunov function 


A 1 1 1 l- 2 

Vo(x1,x2,6) = Vi + <2 = <2i + 2 + -07T 16. (53.30) 
2 2, 2 2 
Because 6 = —6, the derivative of V> is 
gr 
V2 =-a2z{—o23 +l, 2)! aay ale 6-16 
a? 
Oxy 

4 ale) 

= cz} = 02z4 +67! (r E — | [2 _ i) : (53.31) 
1 


The only way to eliminate the unknown parameter error 0 is to select the update law 


iS se 0 O 
6 =Tt2(x, 0) =T E — | | =f (a _ sa 021) ? (53.32) 
1 


Ox 1 22 


Then V2 is nonpositive: 
Y= —cz7 _ o2z3, (53.33) 


which means that the global stability of z = 0, 6 = 0 is achieved. Moreover, by applying either the LaSalle 

or the Barbalat argument mentioned in Design A, we prove that z(t) > 0 as t > oo. Finally, from 

Equation 53.23, it follows that the equilibrium x = 0, 6 =0is globally stable and x(t) > 0as t > oo. 
The crucial property of the control law in Design B is that it incorporates the v2-term (Equation 


53.28) which is proportional to 6 and compensates for the effect of parameter estimation transients on 
the coordinate change (Equation 53.23). It is this departure from the certainty equivalence principle that 
makes the adaptive stabilization possible for systems with nonlinearities of arbitrary growth. 

By comparing Equation 53.32 with Equation 53.19, we note that the first term pz, is the potential 
update law for the z)-system. The functions 


t1(x1) = 921, (53.34) 
Oa, 


12(x1,X2, 0) = ta (x1) — Ox, 02 (53.35) 
1 


are referred to as the tuning functions, because of their role as potential update laws for intermediate 
systems in the backstepping procedure. 
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53.2.1.3 Design C 


The system 
iy = x2 + (x1), 
x2 = X3, (53.36) 
x3 =u 


A 


is obtained by augmenting system (Equation 53.22) with an integrator. The control law a(x), x2, 9) 
designed in (Equation 53.24) can no longer be directly applied because x3 is a state and not a control 
input. We “step back” through the integrator +3 = u and design the control law for the actual input u. 
However, we keep the stabilizing function a2 and use it to define the third error coordinate 


Z3 = X3 — 2 (x1,X2, 8). (53.37) 


The parameter update law (Equation 53.32) will have to be modified with an additional z3-term. Instead 


of 6 in Equation 53.28, the compensating term v2 will now use the potential update law (Equation 53.35) 
for the system (Equation 53.29): 


mn O A 
va(x1,%2, 6) = Sp Pee). (53.38) 


Hence, the role of the tuning function 12 is to substitute for the actual update law in the compensation of 
the effects of parameter estimation transients. 

With Equations 53.23, 53.35, 53.37, and 53.38, and the stabilizing function a2 in Equation 53.24, we 
have 


Zz c 1 4 e 

1 —Cl 1 A 

eae ze 6+ stl. 53.39 

3] | —1 ee B =a OD: i z+ Oo (T't2 — 9) 
Oxy 00 


This system differs from the error system (Equation 53.29) only in its last term. Likewise, instead of 
the Lyapunov inequality (Equation 53.33) we have 


. Ou A x A 
Y= czy C225 + 2223 + Z2 a (Tt —9)+ 6 (1 —r7'6), (53.40) 
Differentiating Equation 53.37, we obtain 


7 Oa T Oa2 OX) 4 
= | 8) —9 
ea x (= ® Ox2 “8 00 


Oa TA 0a Oa A Oa Tx 
=u- — ) —9 0. 53.41 
vs Ox (x ee ) Ox2 Si faye) Oxy 2 ( ) 


We now stabilize the (zi, z2,2z3)-system (Equation 53.39 and 53.41 with respect to the Lyapunov 
function 


A 1 
V3(x, 0) =V>-4 Ze = 


Lito lor-ig 
5 zt sat st 5616. (53.42) 


Its derivative along Equations 53.39 and 53.41 is 


: oa RB Ou ~\ OO Oa + 
V3 = —Cz7 — zt ot —0)+23 E a (x + 9°) = iy ee i 
00 Oxy Ox2 00 


2 0 am 
+ 6F (x — 923 rs) f (53.43) 
1 
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Again we must eliminate the unknown parameter error 6 from V3. For this we must choose the update 
law as 


Oa, Oa 7 


x A Ou 
8 = P13(x1, x2, x3,0) =T ( t 7923) =I |g, Qs || 22]. (53.44) 
Ox] Oxy Ox) 


Upon inspection of the bracketed terms in V3, we pick the control law: 


A Oa A Oa 
U = 03(x1, X2,X3,9) = —Z2 — 6323 + 2 (x2 + 9%) + x3 + v3. (53.45) 
Oxy Ox2 


The compensation term v3 is yet to be chosen. Substituting Equation 53.45 into Equation 53.43, we obtain 


: da A OQ * 
V3= C1 Zi C225 0323 + Z = (Tt, — 9) + 23 (vs a i). (53.46) 


From this expression it is clear that v3 should cancel 02 6 Tn order to cancel the cross-term Z2 a (Pt2 — 6) 
with v3, it appears that we would need to divide by z3. However, the variable z3 might take a zero value 
during the transient, and should be regulated to zero to accomplish the control objective. We resolve this 
difficulty by noting that 


6—Tt, =6—-f134+Tt3—-Tp 


R Ou. 
=6—Tt3 -T— 23, (53.47) 
Oxy 
so that V3 in Equation 53.46 is rewritten as 
. Ou Oa, Oa 
V3= czy 0225 c3Z3 + Z3 ( ie l't3+ at ieee 021) : (53.48) 
08 06 Ox) 


From Equation 53.48, the choice of v3 is immediate: 


A Oa Oa, Oa 
V3(X1,X2, x3, 0) = —-I't3 i gpa (53.49) 
00 06 Oxy 
The resulting V3 is 
V3= —c1Z} _ c2z5 _ €323, (53.50) 


which guarantees that the equilibrium x = 0, 6=60 is globally stable, and x(t) > 0 as t > oo. The 
Lyapunov design leading to Equation 53.48 is effective but does not reveal the stabilization mechanism. 
To provide further insight we write the (z), Z2, z3)-system (Equations 53.39 and 53.41) with u given in 
Equation 53.45 but with v3 yet to be selected: 


0) 0 
Z —C] 1 0 Z] Oa, Oay A 
l=|-1 -a 1 |/m)+|—3,° 16+] 9 @e—9) |. (53.51) 
23 0 —-1l ~e3]| Lz3 0 dat 
= = if V3 — ta l't3 
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Oo2 
08 
equation. By substituting Equation 53.47, we note that a (tz — @) has z3 as a factor and absorb it into 
the “system matrix”: 


oo (Pt) — 4) in the second 


While v3 can cancel the matched term 6, it cannot cancel the term 


T 
er | 0 0 
Z1 Z1 Oa, lies 0 
a en 14 re a|+|— 5, |o+ a _ (53.52) 
23 0 1 00 X] Z3 Oa2 r VR= 2 rts 
= =—¢; 5 
xX) 
Now v3 in Equation 53.49 yields 
~c 1 0 gr 
2) Oa, _ Oo 71 Oo pl 
»|=-|-! —C2 1+ a Vom © aft ra 0. (53.53) 
a 2 ~c3 ca PE 
a0 Ox Ax, 
This choice, which places the term — a roe ¢ at the (2,3) position in the system matrix, achieves skew- 


symmetry with its positive image above the diagonal. What could not be achieved by pursuing a linear-like 
form was achieved by designing a nonlinear system where the nonlinearities are “balanced” rather than 
cancelled. 


53.2.2 General Recursive Design Procedure 


A systematic backstepping design with tuning functions has been developed for the class of nonlinear 
systems transformable into the parametric strict-feedback form: 


iy = x2 + G1 (x1)"9, 


ky = x3 + 2(x1, x2)", 


(53.54) 
Xn-1 = Xn + @n—1(X1; tee iia) ®, 
Xn = B(x)u+ On(x)"6, 
Y=, 
where B and 
F(x) = [¢1(x1); 2(x1, x2), sey Gn(x)] (53.55) 


are smooth nonlinear functions, and B(x) 4 0, Vx € IR”. (Broader classes of systems that can be controlled 
using adaptive backstepping are listed in Section 53.5.) 
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TABLE 53.1 Summary of the Tuning Functions Design for Tracking 


z=xji—ye) - 5-1, (53.56) 
i—1 ae Oaj—1 Oaj-1 (k 
(xi, 6, 96°?) = —z;-1 — cz; — w 4 > ( a Xe 4 way ' E Vis (53.57) 
k=1 *k Oy: 
(i-1) Oouj—1 = Ou 
vi(X, 9,52?) = +——-T 4 — Dwizp, (53.58) 
See a du es 
(%,6,99-D) =z, 2, 
TX 9H") = G-1 + Wiz, (53.59) 
i-1 
ia A oo (Ge Oaj_— 
wi, 6,70 ee : Oks (53.60) 
OX 


k=1 
i=1..n H=(,....%), FO=OnIn..7). 


Adaptive Control Law: 


_ 4 <(n—L)) 4 (0) 
“= XC) [ontx, 0, Vr + yr | ; (53.61) 


Parameter Update Law: 


6 =Tty(x, 6,70) = rw. (53.62) 


é fs A A A 
Note: For notational convenience we define zp = 0, a9 = 0, to = 0. 


The general design summarized in Table 53.1 achieves asymptotic tracking, that is, the output y = x, 
of the system (Equation 53.54) is forced to asymptotically track the reference output y,(t) whose first n 
derivatives are assumed to be known, bounded, and piecewise continuous. 

The closed-loop system has the form 


z=A,(z,6,t)z + W(z, 6, t)"6, (53.63) 
6=rw(z,6, t)z, (53.64) 
where 
—c) 1 0 0 
=1 —C2 1+ 023 Are O2n 
A,(z,4,t)=| 0 —-l-o3 ©. - (53.65) 
1+ On—1,n 
0 —92n cs Ses On-1,n —Cn 
and 
A OOj—1 
ojx (x, 8) = —— Tw. (53.66) 
fale) 


Because of the skew-symmetry of the off-diagonal part of the matrix Az, it is easy to see that the Lyapunov 
function 


1 Lies bs 
V, = -z'z+-6'r 16 (53.67) 
2 2 
has the derivative 


n 
Vn =— > eazy, (53.68) 
k=1 
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which guarantees that the equilibrium z = 0, 6 =0is globally stable and z(t) — 0 as t + oo. This means, 
in particular, that the system state and the control input are bounded and asymptotic tracking is achieved: 
lim} 90 [y(t) = yr(t)] =0. 

To help understand how the control design of Table 53.1 leads to the closed-loop system (Equations 
53.63 through 53.66), we provide an interpretation of the matrix A; for n = 5: 


—Cj 1 0 0 0 0 
—1 —C2 1 0 023 024 025 
A,= —-l -c 1 +10 —o3 034 035]. (53.69) 
—-1l ~-c4 1 0 —024 —034 045 
-1l 65 0 —025 —035 —045 


If the parameters were known, 6 = 8, in which case we would not use adaptation, I’ = 0, the stabilizing 
functions (Equation 53.57) would be implemented with v; = 0, and hence o;; = 0. Then Az would be 


just the above constant tridiagonal asymptotically stable matrix. When the parameters are unknown, we 
use I’ > 0 and, owing to the change of variable zj = x; — yd 
_ ae 6= pe) FikZk appears. The term vj = — yy OikZk — eo OxiZ in the stabilizing function 


(Equation 53.57) is crucial in compensating the effect of 8. The ojx-terms above the diagonal in Equation 


— aj—1, in each of the zj-equations a term 


53.69 come from 0, Their skew-symmetric negative images come from feedback vj. 

It can be shown that the resulting closed-loop system (Equations 53.63 and 53.64), as well as each 
intermediate system, has a strict passivity property from 6 as the input to 1; as the output. The loop 
around this operator is closed (see Figure 53.1) with the vector integrator with gain I’, which is a passive 
block. It follows from passivity theory that this feedback connection of one strictly passive and one passive 
block is globally stable. 


53.3 Modular Design 


In the tuning functions design, the controller and the identifier are derived in an interlaced fashion. This 
interlacing led to considerable controller complexity and inflexibility in the choice of the update law. 

It is not hard to extend various standard identifiers for linear systems to nonlinear systems. It is 
therefore desirable to have adaptive designs where the controller can be combined with different identifiers 
(gradient, least-squares, passivity based, etc.). We refer to such adaptive designs as modular. 

In nonlinear systems it is not a good idea to connect a good identifier with a controller which is good 
when the parameter is known (a “certainty equivalence” controller). To illustrate this, let us consider the 


FIGURE 53.1 The closed-loop adaptive system (Equations 53.29 and 53.32). 
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error system 
x = —x+ —(x)0 (53.70) 


obtained by applying a certainty equivalence controller u = —x — ¢(x)@ to the scalar system x =u+ 
¢(x)0. The parameter estimators commonly used in adaptive linear control generate bounded estimates 
6(t) with convergence rates not faster than exponential. Suppose that 6(t) = e~* and g(x) = x°, which, 
upon substitution in Equation 53.70, gives 


k= —xtx3 07! (53.71) 
For initial conditions |x| > 3, the system (Equation 53.71) is not only unstable but its solution escapes 
to infinity in finite time: 


1 
x(t) > oo ast——lIn =. (53.72) 
3 x? —3/2 


From this example we conclude that for nonlinear systems we need stronger controllers which prevent 
unbounded behavior caused by 9. 


53.3.1 Controller Design 


We strengthen the controller for the preceding example, u = —x — ¢(x)6, with a nonlinear damping term 
—(x)*x, that is, u = —x — e(x)6 — e(x)*x. With this stronger controller, the closed-loop system is 


x= —x — e(x)?x+ o(x)0. (53.73) 


To see that x is bounded whenever 6 is, we consider the Lyapunov function V = 5x, Its derivative along 
the solutions of Equation 53.73 is 


1 
4 


2 
V= —x? — e(x)*x? 1 xo(x)0 = x? | woos i] oO 


les) 
s+ 28. (53.74) 


From this inequality it is clear that |x(t)| will not grow larger than 1/6(£)|, because then V becomes 
negative and V = bx? decreases. Thanks to the nonlinear damping, the boundedness of 6(t) guarantees 
that x(t) is bounded. 

To show how nonlinear damping is incorporated into a higher-order backstepping design, we consider 
the system 

Xy = x2 + 9(x1)"9, 


‘ (53.75) 
x2 =U. 


Viewing x2 as a control input, we first design a control law 04 (x1, 9) to guarantee that the state x; 
in x) =x2.+ (x1)"0 is bounded whenever 6 is bounded. In the first stabilizing function we include a 
nonlinear damping term* —k | (x1) (2x1: 


ony (x1, 6) = —eyx1 — (1) 76 — kr p(xi)I2a1, 1, K1 > 0. (53.76) 


Then we define the error variable z2 = x2 — 04 (x1, 9), and for uniformity denote z; = x;. The first 
equation is now 


Zjy = -C1Z — K1|9|7z1 + p06 + zp. (53.77) 


* The Euclidian norm of a vector v is denoted as |v| = V vl v. 
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If z2 were zero, the Lyapunov function V; = Sey would have the derivative 


14 
Z 
G21 Te 


7 2 2,2 TA 2 
Vy = -c1z7 — Kil gl*z7 +219 9 = —czj — Ky 


so that z; would be bounded whenever 6 is bounded. With Z2 #0 we have 
: ih, Dee 
Vy < -cz7 + Gel +2122. (53.79) 
Ky 


Differentiating x2 = z2 + 01 (x1, 6), the second equation in Equation 53.75 yields 


. ‘ ‘ Oa, T Oay A 
Zs = 8) 6. 53.80 
2 = xX2- Oy =U Ox, (x2 Q aa ( ) 


The derivative of the Lyapunov function 
Up the 
V2 Vi 52 - | | (53.81) 


along the solutions of Equations 53.77 and 53.80 is 


V2 <-czp+ =r +2n+z [- _ (x2 + 96) - ~ i 
< —cz2 + Pa |6|2 +2 E +2z,- ~~ (x2 + 976) (= gp 6+ “ i) ‘ (53.82) 
We note that now, in addition to the 6-dependent disturbance term oat ' 6, we also have a §-dependent 
disturbance 6. No such term appeared in the scalar system (Equation 53.73). We now use nonlinear 
damping terms —k2 oa of Z and —go | oo 2 [ Z, to counteract the effects of both 6 and 6: 
da, |? Oa, 7 ? Oay ae 
u= —Z| — C2Z2 — K2 an Q| 22-22 aA Z244 Ox, (x2 + 8) ; (53.83) 


where cp, K2, g2 > 0. Upon completing the squares as in Equation 53.78, we obtain 


. 1 1 ~ 1 aA 
V2 <-c1z? —azet+ + 6? + 6/7, 53.84 
2S 1 2zy — 0225 te ie |0| ag) | ( ) 
which means that the state of the error system 
—c) —k2 l@l? 1 g! 0 
2 2 i 

t= do, |? doiT) 1z+] aay 8+] da, [8 (53.85) 

—1 C2 K2 x aS gt ——_ 

Ox) 0 Oxy a6 


is bounded whenever the disturbance inputs 6 and 6 are bounded. Moreover, since V2 is quadratic in z, 
see Equation 53.81, we can use Equation 53.84 to show that the boundedness of z is guaranteed also when 
0 is square-integrable but not bounded. This observation is crucial for the modular design with passive 
identifiers where 6 cannot be a priori guaranteed to be bounded. 


The recursive controller design for the parametric strict-feedback systems (Equation 53.54) is summa- 
rized in Table 53.2. 
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TABLE 53.2 Summary of the Controller Design in the Modular Approach 


y=x—ye) —ai-1, (53.86) 
1 = Oj—1 Oaji-1 (k 
(Xj, 6, i y= Zi-1 — CiZi w)64 ( ie Xeai4 ay ' SiZis (53.87) 
k=1 k Oy: 
i-1 Pre 
wi (6,7) = 3 - Oh (53.88) 
a Oxf 
2 
Oaj— 
si(%, 6,9) = Kilwil? +g:| "|, (53.89) 
(Slush MSCs We = Onis uy 
Adaptive Control Law: 
=aG) ~ [oan(x8 6,9") +4]. (53.90) 


Controller Module Guarantees: 


If 6€Lo0 and bE Ly or Loo thenx € Lo. 


‘ F A A 
Note: For notational convenience we define zp = 0, a9 = 0. 


Comparing the expression for the stabilizing function (Equation 53.87) in the modular design with the 
expression (Equation 53.57) for the tuning functions design we see that the difference is in the second 
lines. While the stabilization in the tuning functions design is achieved with the terms v;, in the modular 
design this is accomplished with the nonlinear damping term —s;z;, where 


2 
yn NT Gat OQj-1 a 
si(Xi, 6,79) = Kil wil? " (53.91) 
00 
The resulting error system is 
z= A,(z,6,t)z + Wiz, 6, t)164 Q(z, 6, £)76, (53.92) 
where A,, W, and Q are 
—C, — $j 1 0 0 
—1 —C2 — $2 1 
Az(z,6,t) = 0 =4 Tete 0 > 
7 : 
0 0 -1l ~-y- Sp 
0 
wi Oa) 
: W> : 80 
W822) S15: OCB Si) oa (53.93) 
Pa A0n-1 


53-14 Control System Advanced Methods 


Since the controller module guarantees that x is bounded whenever 0 is bounded and 6 is either 
bounded or square-integrable, then we need identifiers which independently guarantee these properties. 
Both the boundedness and the square-integrability requirements for 6 are essentially conditions which 
limit the speed of adaptation, and only one of them needs to be satisfied. The modular design needs slow 
adaptation because the controller does not cancel the effect of 8, as was the case in the tuning functions 
design. 

In addition to boundedness of x(t), our goal is to achieve asymptotic tracking, that is, to regulate z(t) 
to zero. With z and 6 bounded, it is not hard to prove that z(t) > 0 provided 


W(2(t), (),t)'0(t) > 0 and @(t) > 0. 


Let us factor the regressor matrix W, using Equations 53.93, 53.88, and 53.55 as 


1 0 see 0 
a 
J 4 
W(z,6,t)? = ve F(x)" 2 N(z,6, t)F(x)". (53.94) 
; 0 
= OOn—1 = OOn—1 l 
Oxy OXn-1 


Since the matrix N(z, 6, t) is invertible, the tracking condition W(z(t), (t), t)26(t) > 0 becomes 
F(x(t))0(t) > 0. 
In the next two subsections we develop identifiers for the general parametric model 


x = f(x,u) + F(x, u)"6. (53.95) 


The parametric strict-feedback system (Equation 53.54) is a special case of this model with F(x, u) given 
by Equation 53.55 and f(x, u) = [x2,..-5Xn Bo(x)u]!. 

Before we present the design of identifiers, we summarize the properties required from the identifier 
module: 


1. 8€ Lo and BEL, or Loos 
2. Ifx€ Loo then F(x(t))'6(t) > 0 and A(t) > 0. 


We present two types of identifiers: the passive identifier and the swapping identifier. 


53.3.2 Passive Identifier 


For the parametric model (Equation 53.95), we implement, as shown in Figure 53.2, the “observer” 
— [40 = KF (x, u) PCS, uP] (&—x) +f (x, u) + F(x, u)"6, (53.96) 


where ) > 0 and Apo is an arbitrary constant matrix such that 


PAjp+AjP=-I, P=P'>0. (53.97) 
By direct substitution it can be seen that the observer error 
€=x-x (53.98) 


is governed by 
g2 [40 = KP (x4) TPO uP] e+ F(x, u)"6, (53.99) 


Adaptive Nonlinear Control 53-15 


x=f+F'o 


& = (Ag — AET EP) (&- x) + f+ F6 a 


6 6 
/ T FP 


FIGURE 53.2 The passive identifier. 


The observer error system (Equation 53.99) has a strict passivity property from the input 6 to the output 
F(x, u)Pe. A standard result of passivity theory is that the equilibrium 6 = 0, « = 0 of the negative feedback 
connection of one strictly passive and one passive system is globally stable. Using integral feedback such 
a connection as in Figure 53.3 can be formed. This suggests the use of the following update law: 


6=LF(x,u)Pe. T=P'>o. (53.100) 
To analyze the stability properties of the passive identifier we use the Lyapunov function 
V=0'T '0+e'Pe. (53.101) 


After uncomplicated calculations, its derivative can be shown to satisfy 


. h oA 
V< ele [07 (53.102) 
(1)? 

This guarantees the boundedness of 6 and ¢, even when 2 = 0. However, 6 cannot be shown to be bounded 
(unless x and u are known to be bounded). Instead, for the passive identifier one can show that Dis square- 
integrable. For this we must use ) > 0, that is, we rely on the nonlinear damping term —F(x, u)! F(x, u)P 
in the observer. The boundedness of 6 and the square-integrability of 6 imply (cf. Table 53.2) that x is 
bounded. 

To prove the tracking, we need to show that the identifier guarantees that, whenever x is bounded, 
F(x(t))"6(t) > 0 and 6(t) > 0. Both properties are established by Barbalat’s lemma. The latter property 
can easily be shown to follow from the square-integrability of 0. The regulation of F(x)'6 to zero follows 
upon showing that both e(t) and €(t) converge to zero. While the convergence of €(t) follows by deducing 
its square-integrability from Equation 53.102, the convergence of €(t) follows from the fact that its integral, 
ihe €(t) dt = €(00) — €(0) = —€(0), exists. 


a 


(99) FPe 


Ep 


FIGURE 53.3 Negative feedback connection of the strictly passive system (Equation 53.99) with the passive 


system c : 
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53.3.3 Swapping Identifier 


For the parametric model (Equation 53.95), we implement two filters, 


or [40 — RFC, a) POY, uP] OT + F(x, u)", (53.103) 


oe [40 — VF (x, w) F(x, uP] (Qo — x) —flx, w), (53.104) 
where i > 0 and Ap is as defined in Equation 53.97. The estimation error, 


€=x+Q2)—Q76. (53.105) 


can be written in the form 
€=276+4+¢, (53.106) 


where € 2x4 Qo — 276 decays exponentially because it is governed by 
= [40 — XF(x, uw)" F(x, uP] é, (53.107) 


The filters (Equations 53.103 and 53.104) have converted the dynamic model (Equation 53.95) into the 
linear static parametric model (Equation 53.106) to which we can apply standard estimation algorithms. 
As our update law we will employ either the gradient 


Qe 


rr ee eer 53.108 
14 vtr{QTQ} = ( 
or the least-squares algorithm 
A Qe 
§ = [——_____, 
1+ vtr{QTQ} 
Sor (53.109) 
r r, 0)=r(o)' >0, v=o. 
1+ vtr{QTQ} 


By allowing v = 0, we encompass unnormalized gradient and least squares. The complete swapping 
identifier is shown in Figure 53.4. 
The update law normalization, v > 0, and the nonlinear damping, i > 0, are two different means for 


slowing down the identifier in order to guarantee the boundedness and square-integrability of 6. 
For the gradient update law (Equation 53.108), the identifier properties (boundedness of 6 and 6 and 


regulation of F(x)@ and 0) are established via the Lyapunov function 
1- 
V= 50T 10+ ee (53.110) 
whose derivative is 


3 ele 


Vie a, (53.111) 
414 vtr{QTQ} 


The Lyapunov function for the least-squares update law (Equation 53.109) is V = 0'T'(t)~!6 + €Pé. 


53.4 Output Feedback Designs 


For linear systems, a common solution to the output-feedback problem is a stabilizing state-feedback 
controller employing the state estimates from an exponentially converging observer. Unfortunately, 
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x=f+F' 


Qo = (Ag- AFTFP) (Q,-x) +f |2 “Gs 


Q. = (Ag - AFTEP) Q + F >| x 


|, ry TQ 
| bt 
1+vfQ/? 


FIGURE 53.4 The swapping identifier. 


this approach is not applicable to nonlinear systems. Additional difficulties arise when the non- 
linear plant has unknown parameters because adaptive observers, in general, are not exponentially 
convergent. 

These obstacles have been overcome for systems transformable into the output-feedback form: 


x= Ax+t o(y)+ O(y)at+ H o(y)u, xeé IR" 


(53.112) 
y=elx, 
where only the output y is available for measurement, 
0 
, Tn-1 
A=|: , (53.113) 
0 ---0 
0,1(y) Pi) ++ gi) 
o(y) = , Oy)= ; ; (53.114) 
Go,n(y) Pin(y) ace any) 
and the vectors of unknown constant parameters are 
@=laivivisdal x! BH biy24 bol (53.115) 


We make the following assumptions: the sign of b,, is known; the polynomial B(s) = bys +--+ + bis+ 
bo is known to be Hurwitz; and o(y) 4 0 Vy € R. An important restriction is that the nonlinearities p(y) 
and ®(y) are allowed to depend only on the output y. Even when 9 is known, this restriction is needed to 
achieve global stability. 


53-18 Control System Advanced Methods 


TABLE 53.3 State Estimation Filters 


Filters: 
& = Ao§ + ky + o(y), (53.116) 
E=Ape+ &(y), (53.117) 
h=Aodh + eno(y)u. (53.118) 
y=Abh, j=0....m, (53.119) 
Q? = [vins.. eV V0 El. (53.120) 


We define the parameter-dependent state estimate 
R=E+079, (53.121) 


which employs the filters given in Table 53.3, with the vector k = [kj,..., kn]! chosen so that the matrix 
Ap = A— ket is Hurwitz, that is, 


PAg+AjP=-I, P=P'>0. (53.122) 

The state estimation error 
e=x—-X (53.123) 

is readily shown to satisfy 
&= Acs. (53.124) 


The following two expressions for y are instrumental in the backstepping design: 


J=optw'O+e2 (53.125) 
= DinVin +00 + 610+ €2, (53.126) 
where 
0 = 0,1 + &2, (53.127) 
© = [Vin2>Vm—1,23+++>%02» Pay + Eyl’, (53.128) 
® = [0, Vin—125-++>Vo21 Pay + Be]. (53.129) 


Since the states x2,...,X, are not measured, the backstepping design is applied to the system 


J = binVn2 +0 +0104 £9, (53.130) 
Ving =Vmnitt —kivmi, i=2,...,p—1, (53.131) 
Vino = O(Y)U+ Vinjo+1 — Kp Vm,1- (53.132) 


The order of this system is equal to the relative degree of the plant (Equation 53.112). 


53.4.1 Output-Feedback Design with Tuning Functions 


The output-feedback design with tuning functions is summarized in Table 53.4. The resulting error 
system is 
z= A,(z,t)z + Welz, thea + Wo(z,t)' 6 — bin (jr + G1) €10, (53.133) 
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where 
—c) —d) a 0 0 
4 Oa, ? 
bin co — dz ( ) 1+ 03 024 vee 02,6 
oy 
0 —-l- 023 
Az,= . : : 
—024 2 De . Op—2,0 
1+ 09-10 
Oa ey 2 
(53.134) 
and 
1 
Oa, 
y T eT eo T 
W.(z, t) = : , Wol(z,t) = W,(z,t)w —6 (i: + a1) e€1e}. (53.135) 
O0p—1 
Oy 


2 
The nonlinear damping terms —d; (25) in Equation 53.134 are included to counteract the expo- 


nentially decaying state estimation error ¢2. The variable 6 is an estimate of 9 = 1/bm. 


TABLE 53.4 Output-Feedback Tuning Functions Design 


Z=y—yr (53.136) 
~ (i-1) : 
Zi =Vm,i — Or (Fe ne ee + (53.137) 
a= 0a), a= (4 t d;) Z1 — Wo 616, (53.138) 
A Ba, \? : Oa; \; Oa 
a2 = —bmz, E + do ( 5) )p t (5. t me) + = Tt2+ 2, (53.139) 
eee ee (=) Be ( (=), Ot j 
1 1 1 1 oy 1 rt 06 2 
1 
Oorj—1 Wo Oj 1 Ooi . 
eS oe SBR 1] 3ces th (53.140) 
a > a | By 17h 


oy 0§ d= 
ivl m+i-1 
Ooj-1 Oaj— 
p> =e + ki¥ma 4 Fn (kia + yj) (53.141) 
j=l OVr j=l J 
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TABLE 53.4 (Continued) Output-Feedback Tuning Functions Design 


ty = (w— 6 (jr + %1) €1) 21, (53.142) 
Oaj— 
GSy4= om. 2 talb (53.143) 
dy 
Adaptive Control Law: 
ae (ap ee ay”) (53.144) 
o(y) 
Parameter Update Laws: 
§=Tty (53.145) 
é = —y sgn(bm) (ir + a1) Zy. (53.146) 


TABLE 53.5 Output-Feedback Controller in the Modular Design 


Z=y—Yr (53.147 
1 @-1) ’ 
Zi =Vm,i — >—Ve OF. OES Ds 450 5 Pe (53.148 
m 
b 1 x 
a= sgn(bm) (c1 t s1) Z+>—-Q1, A =—9 a6, (53.149 
Sm bm 
Q2= bmz\ (c 52) z2 +82, (53.150 
oj = —z-1-(c9+si))at+Bi, 1=3,...,0, (53.151 
ait Lal) 4 Ooi nt i 1 Ooi el 
B= (9 +076) + F=* (Aok + ky +6) + Fe* (408 + ®) 
i-1 m+i-1 
Oaj-1 j Oaj—1 
te 4 Kiva 4 Fe (hy + Aya. (53.152 
j=l OV; j=l I 
1 Z 
sp=di t+ |@+ — (w+ai)ei] , (53.153 
bm 
2, 
sgQ=d. (a) + an ze ‘ peat} yre (53.154 
DAD: TK 1€] vr §2 x x || + 
dy dy Saag. ga 
Oa; 2 Ou; daj;-;! 1 : 
sai ( <1) + ej | | tgif sy Mel], i=3,...,0. (63.155 
oy oy 06 b2, 
Adaptive Control Law: 
1 1 ”) 
= Qp — V, t+ = ; (53.156) 
o(y) ( p mo+l be 
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53.4.2 Output-Feedback Modular Design 


In addition to sgn(b,,), in the modular design we assume that a positive constant ¢), is known such that 
|Om| 2 Sm: 
The complete design of the control law is summarized in Table 53.5. The resulting error system is 


z= AX(z, t)z + Welz, tee + Welz, t)7O+ Q(z, t)"6, (53.157) 
where 
b 
lem 4st) bin 0 see 0 
m 
—bin —(2+s2) 1 

Ax(z,t) = ; (53.158) 

—l 0 

1 

0 0 -1 —(cp)+5p) 


1 he 
dcr = at +zel 
oy «(> aT a 
W.(z, t) = ‘ Ws (z,t) = 2 GOD: T > (53.159) 
: 0 
OOp—1 
a : 
y _ OOp—1 T 
dy 
0 
Ol 1 
Ba = pret 
e 00 ib? 
Q(z, t)) = . ; (53.160) 
Ope as Eas Co Der 
00 b2 
53.4.2.1 Passive Identifier 
For the parametric model (Equation 53.125), we introduce the scalar observer 
¥=—(co+kolol?) (Fy) too +016. (53.161) 
The observer error 
e=y-y (53.162) 
is governed by 
€=—(cot+kolal’?)e +o! 0+e2. (53.163) 
The parameter update law is 
6 = Proj {Pac}, Pm (O)sgnbm > Sm (53.164) 


—yT 
bn r=Il">0 


where the projection operator is employed to guarantee that [bm(t)| > Sm > 0, Vt > 0. 
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53.4.2.2 Swapping Identifier 
The estimation error 
e=y—£,-276 (53.165) 


satisfies the following equation linear in the parameter error: 
e=2764+e). (53.166) 


The update law for 0 is either the gradient: 


r r=r'so 53.167 
Tp va 2 > ( ) 


‘5 Qié bm(0)sgnbm > Sm 
0 = Proj 
v>0 


or the least squares: 
x . Qye ry 
6= Proj {rs I > by (0)sgnby, > Sm 
(53.168) 


oor - r(0) = 10) > 0 
1+v]Q,|2.”> v>0. 


53.5 Extensions 


Adaptive nonlinear control designs presented in the preceding sections are applicable to classes of non- 
linear systems broader than the parametric strict-feedback systems (Equation 53.54). 


53.5.1 Pure-Feedback Systems 


Xi = Xit4.+ (x1, . ba ,Xi+1) 9, t= 1,2.gn= 1, 
(53.169) 
Sin = (Bol) + B(x)™6) w+ gol) + onlx)"6, 
where (0) = 0, 1 (0) =--- = @,(0) = 0, and Bo(0) 4 0. Because of the dependence of ; on x;+1, 


the regulation or tracking for pure-feedback systems is, in general, not global, even when 0 is known. 


53.5.2 Unknown Virtual Control Coefficients 


ij = bixin, + Oil, ...,x;)19, i=1l,...,n—-1, 
(53.170) 
Xn = bnB(x)U+ On(x1,---%n)"8, 


where, in addition to the unknown vector 9, the constant coefficients b; are also unknown. The unknown 
b;-coefficients are frequent in applications ranging from electric motors to flight dynamics. The signs of 
bj, i=1,...,n, are assumed to be known. In the tuning functions design, in addition to estimating b;, we 
also estimate its inverse 9; = 1/b;. In the modular design we assume that in addition to sgnbj, a positive 
constant ¢; is known such that [bj | > cj. Then, instead of estimating 0; = 1/b;, we use the inverse of the 
estimate bj, that is, 1 i b;, where b;(t) is kept away from zero by using parameter projection. 
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53.5.3 Multi-Input Systems 


Xj = BX) Xi41 + O1(Kj)'9, i=1,...,n-1, 
(53.171) 
Xn = By(X)u+ On(X)"9, 


, = T = : = 
where X; is a vj-vector, Vv} < v2 < +--+ < vy, Xi = [xy, vs pth ,X = X,. and the matrices B;(X;) have 


full rank for all X; € RU! “i, The input u is a v,-vector. The matrices B; can be allowed to be unknown, 
provided they are constant and positive definite. 


53.5.4 Block Strict-Feedback Systems 


i = xi41 t Oi(21,..-5 Xb... -5C4)'O, i= 1; 2245 0 1, 


Xp = B(x, 6)u+ p(x, 6)", (53.172) 
bi = Pio) + Oi%,6;)"0, I=1,...50, 
with the following notation: x; = [x,...,x;]',t = Ee si89 eT]? ,x =Xp, and ¢ = Cp. Each ¢j-subsystem 


of Equation 53.172 is assumed to be bounded-input bounded-state (BIBS) stable with respect to the input 
(Xj, ¢;-1). For this class of systems it is quite simple to modify the procedure in Tables 53.1 and 53.2. Because 
of the dependence of ¢; on ¢;, the stabilizing function a; is augmented by the term + Set | Deo, 


’ T 
and the regressor w; is augmented by — Pas ®; (Se ) ; 


53.5.5 Partial State-Feedback Systems 
In many physical systems there are unmeasured states as in the output-feedback form (Equation 53.112), 
but there are also states other than the output y = x, that are measured. An example of such a system is 
i = x2 + 1(x1)'8, 
ky = x3 + Go2(x1, x2)", 
3 = x4 + 3(x1,x2)"0, 
deg = x5 + 4x1, x2)'O, 
ks = ut @5(x1,%2,%5) "0. 
The states x3 and xq are assumed not to be measured. To apply the adaptive backstepping designs 
presented in this chapter, we combine the state-feedback techniques with the output-feedback techniques. 


The subsystem (x2, x3, x4) is in the output-feedback form with x2 as a measured output; hence we employ 
a state estimator for (x2, x3, x4) using the filters introduced in Section 53.4. 


53.6 For Further Information 


Here we have briefly surveyed representative results in adaptive nonlinear control, a research area that 
exhibited rapid growth in the 1990s and continues to be active today. 

The first adaptive backstepping design was developed by Kanellakopoulos et al. [5]. Its overparametriza- 
tion was removed by the tuning functions design of Krsti¢ et al. [8]. Possibilities for extending the class 
of systems in [5] were studied by Seto et al. [17]. 

Among the early estimation-based results are Sastry and Isidori [16], Pomet and Praly [13], and so on. 
They were surveyed in Praly et al. [14]. All these designs involve some growth conditions. The modular 
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approach of Krsti¢ and Kokotovi¢ [10] removed the growth conditions and achieved a complete separation 
of the controller and the identifier. 

One of the first output-feedback designs was proposed by Marino and Tomei [11]. Kanellakopoulos 
et al. [6] presented a solution to the partial state-feedback problem. Subsequent efforts in adaptive non- 
linear control focused on broadening the class of nonlinear systems for which adaptive controllers are 
available. Jiang and Pomet [4] developed a design for nonholonomic systems using the tuning functions 
technique. Khalil [7] developed semiglobal output feedback designs for a class which includes some 
systems not transformable into the output feedback form. 

For a complete and pedagogical presentation of adaptive nonlinear control the reader is referred to the 
text “Nonlinear and Adaptive Control Design” by Krsti¢ et al. [9]. The book introduces backstepping and 
illustrates it with numerous applications (including jet engine, automotive suspension, aircraft wing rock, 
robotic manipulator, and magnetic levitation). It contains the details of methods surveyed here and their 
extensions. It also covers several important topics not mentioned here. Among them is the systematic 
improvement of transient performance. It also shows the advantages of applying adaptive backstepping 
to linear systems. 

Hundreds of papers have been written on the subject of adaptive nonlinear control and backstepping 
design over the last decade and a half. A detailed citation survey is beyond the scope of this chapter, but we 
mention some of the key groups of results. Adaptive nonlinear controllers have been developed for the class 
of strict-feedforward systems, some classes of stochastic nonlinear systems, and some classes of nonlinear 
systems involving time delays. Robustness of adaptive backstepping controllers to disturbances and 
unmodeled dynamics has been studied, as well as parameter convergence and identifiability. Extensions to 
discrete-time nonlinear systems have been developed. Some classes of nonlinearly parametrized problems 
have also been considered. The fundamental problem of asymptotic behavior of parameter estimates 
in the absence of persistency of excitation has also been addressed, with surprising results—estimates 
may converge to destabilizing frozen values from large sets of initial conditions. Decentralized forms 
of adaptive backstepping have also been developed. Numerous control application results have been 
reported using adaptive backstepping methods, from automotive, aerospace, and underwater vehicles, to 
biochemical systems, magnetic levitation, HVAC, and so on. 

A number of books have been inspired by the design frameworks introduced in [9]. We mention only a 
few of the books. Marino and Tomei [12] and Qu [15] presented additional or alternative recursive design 
techniques for adaptive and robust nonlinear control. Spooner et al. [18] combined the methods presented 
this chapter with neural and fuzzy approximation techniques. French et al. [3] studied the performance 
properties of adaptive backstepping controllers employing neural networks. Dawson et al. [2] presented a 
comprehensive methodology for adaptive nonlinear control of electric machines based on backstepping 
methods. Astolfi et al. [1] developed the idea of system immersion and manifold invariance, leading to 
modular rather than classical Lyapunov schemes. 
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54.1 Introduction 


Intelligent control, the discipline where control algorithms are developed by emulating certain 
characteristics of intelligent biological systems, is an emerging area of control that is being fueled by 
the advancements in computing technology (Antsaklis and Passino, 1993; Passino, 2005). For instance, 
software development and validation tools for expert systems (computer programs that emulate the actions 
of a human who is proficient at some task) are being used to construct “expert controllers” that seek to 
automate the actions of a human operator who controls a system. Other knowledge-based systems such 
as fuzzy systems (rule-based systems that use fuzzy logic for knowledge representation and inference) and 
planning systems (that emulate human planning activities) are being used in a similar manner to auto- 
mate the perceptual, cognitive (deductive and inductive), and action-taking characteristics of humans 
who perform control tasks. Artificial neural networks emulate biological neural networks and have been 
used to (1) learn how to control systems by observing the way that a human performs a control task, and 
(2) learn in an online fashion how to best control a system by taking control actions, rating the quality 
of the responses achieved when these actions are used, then adjusting the recipe used for generating 
control actions so that the response of the system improves. Genetic algorithms are being used to evolve 
controllers via off-line computer-aided-design of control systems or in an online fashion by maintaining 
a population of controllers and using survival of the fittest principles where “fittest” is defined by the 
quality of the response achieved by the controller. 

From these examples we see that computing technology is driving the development of the field of control 
by providing alternative strategies for the functionality and implementation of controllers for dynamical 
systems. In fact, there is a trend in the field of control to integrate the functions of intelligent systems, 
such as those listed above, with conventional control systems to form highly “autonomous” systems 
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that have the capability to perform complex control tasks independently with a high degree of success. 
This trend toward the development of intelligent autonomous control systems is gaining momentum as 
control engineers have solved many problems and are naturally seeking control problems where broader 
issues must be taken into consideration and where the full range of capabilities of available computing 
technologies is used. 

The development of such sophisticated controllers does, however, still fit within the conventional 
engineering methodology for the construction of control systems. Mathematical modeling using first 
principles or data from the system, along with heuristics are used. Some intelligent control strategies rely 
more on the use of heuristics (e.g., direct fuzzy control) but others utilize mathematical models in the same 
way that they are used in conventional control (e.g., see the chapter in this section on neural control of 
nonlinear systems), while still others use a combination of mathematical models and heuristics (see, e.g., 
the approaches to fuzzy adaptive control in Passino, 2005). There is a need for systematic methodologies 
for the construction of controllers. Some methodologies for the construction of intelligent controllers are 
quite ad hoc (e.g., for the fuzzy controller) yet often effective since they provide a method and formalism for 
incorporating and representing the nonlinearities that are needed to achieve high-performance control. 
Other methodologies for the construction of intelligent controllers are no more ad hoc than ones for 
conventional control (e.g., for neural and fuzzy adaptive controllers). There is a need for nonlinear 
analysis of stability, controllability, and observability properties. Although there has been significant 
progress recently in stability analysis of fuzzy, neural, and expert control systems, there is need for much 
more work in nonlinear analysis of intelligent control systems, especially in hybrid ones. Simulations and 
experimental evaluations of intelligent control systems are necessary. Comparative analysis of competing 
control strategies (conventional or intelligent) is, as always, important. Engineering cost-benefit analysis 
that involves issues of performance, stability, ease-of-design, lead-time to implementation, complexity of 
implementation, cost, and other issues must be used. 

Overall, while the intelligent control paradigm focuses on biologically motivated (bioinspired) 
approaches (and uses what some call biomimicry), there are sometimes only small differences in the 
behavior of the resulting controllers that are finally implemented (intelligent are not mystical; they are 
simply nonlinear, often adaptive controllers). This is, however, not surprising since there seems to be an 
existing conventional control approach that is analogous to every new intelligent control approach that 
has been introduced. This is illustrated in Table 54.1. It is not surprising then that while there seem to 
be some new concepts growing from the field of intelligent control, there is a crucial role for the control 
engineer and control scientist to play in evaluating and developing the field of intelligent control. For 
more detailed discussions on the relationships between conventional and intelligent control; see Passino 
(2005). 


TABLE 54.1 Analogies between Conventional and Intelligent Control 


Intelligent Control Technique Conventional Control Approach 

Direct fuzzy control Nonlinear control 

Fuzzy adaptive/learning control Adaptive control and identification 

Fuzzy supervisory control Gain-scheduled control, hierarchical control 

Direct expert control Controllers for automata, Petri nets, and other discrete event 
systems 

Planning systems for control Certain types of controllers for discrete event systems, receding 


horizon control of nonlinear systems, model predictive control 


Neural control Adaptive control and identification, optimal control 
Genetic algorithms for computer-aided-design CAD using heuristic optimization techniques, optimal control, 
(CAD) of control systems, controller tuning, receding horizon control, and stochastic adaptive control 


identification 
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In this chapter, we briefly examine the basic techniques of intelligent control, provide an overview of 
intelligent autonomous control, and discuss some advances that have focused on comparative analysis, 
modeling, and nonlinear analysis of intelligent control systems. The intent is only to provide a brief 
introduction to the field of intelligent control and to the next two chapters; the interested reader should 
consult the references provided at the end of the chapter, or the chapters on fuzzy and neural control for 
more details. 


54.2 Intelligent Control Techniques 


Major approaches to intelligent control are outlined below and references are provided in case the reader 
would like to learn more about any one of these approaches. It is important to note that while each of 
the approaches is presented separately, in practice there is a significant amount of work being done to 
determine the best ways to utilize various aspects of each of the approaches in “hybrid” intelligent control 
techniques. For instance, neural and fuzzy control approaches are often combined. In other cases, neural 
networks are trained with genetic algorithms. One can imagine justification for integration of just about 
any permutation of the presented techniques depending on the application at hand. 


54.2.1 Fuzzy Control 


A fuzzy controller can be designed to crudely emulate the human deductive process (i.e., the process 
whereby we successively infer conclusions from our knowledge). As shown in Figure 54.1 the fuzzy con- 
troller consists of four main components. The rule-base holds a set of “IF-THEN” rules that are quantified 
via fuzzy logic and used to represent the knowledge that human experts may have about how to solve a 
problem in their domain of expertise. The fuzzy inference mechanism successively decides what rules are 
most relevant to the current situation and applies the actions indicated by these rules. The fuzzification 
interface converts numeric inputs into a form that the fuzzy inference mechanism can use to determine 
which knowledge in the rule-base is most relevant at the current time. The defuzzification interface com- 
bines the conclusions reached by the fuzzy inference mechanism and provides a numeric value as an 
output. Overall, the fuzzy control design methodology, which primarily involves the specification of the 
rule-base, provides a heuristic technique to construct nonlinear controllers and this is one of its main 
advantages. For more details on direct fuzzy control see the next chapter or Passino and Yurkovich (1998). 

Often it is the case that we have better knowledge about how to control a process such as, information 
on how to tune the controller while it is in operation or how to coordinate the application of different 
controllers based on the operating point of the system. For instance, in aircraft control certain key variables 
are used in the tuning (scheduling) of control laws and fuzzy control provides a unique approach to the 
construction and implementation of such a gain scheduler. In process control, engineers or process 
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FIGURE 54.1 Fuzzy control system. 
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FIGURE 54.2 Rule-based supervisory control. 
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operators often have a significant amount of heuristic expertise on how to tune proportional-integral- 
derivative (PID) controllers while they are in operation and such information can be loaded into the 
rule-base of a “fuzzy PID tuner” and used to make sure that a PID controller is tuned properly at all 
times. In the more general case, we may have knowledge of how to tune and coordinate the application 
of conventional or fuzzy controllers and this can be used in a rule-based supervisor as it is shown in 
Figure 54.2. For more details on fuzzy supervisory control see the next chapter. 

In other fuzzy control approaches, rather than implementing deductive systems, the goal is to imple- 
ment an “inductive system,” that is, one that can learn and generalize from particular examples (e.g., 
examples of how the system is behaving). Such approaches typically fall under the title of “fuzzy learning 
control” or “fuzzy adaptive control.” In one approach, shown in Figure 54.3, called fuzzy model reference 
learning control (FMRLC) (Layne and Passino, 1993), there is a fuzzy controller with a rule-base that 
has no knowledge about how to control the system. A “reference model” with output y,,(t) is used to 
characterize how you would like the closed-loop system to behave (i.e., it holds the performance speci- 
fications). Then, a learning mechanism compares y(t) to ym(t) (i.e., the way that the system is currently 
performing to how you would like it to perform) and decides how to synthesize and tune the fuzzy con- 
troller so that the difference between y(t) and y(t) goes to zero and hence, the performance objectives 
are met. 

Overall, our experiences with the FMRLC indicate that significant advantages may be obtained if 
one can implement a controller that can truly learn from its experiences (while forgetting appropriate 
information) so that when a similar situation is repeatedly encountered the controller already has a 
good idea of how to react. This seems to represent an advance over some adaptive controllers where 
parameters are adapted in a way such that each time the same situation is encountered, some amount of 
(often complete) readaptation must occur no matter how often this situation is encountered (for more 
details on this and other fuzzy learning/adaptive control approaches see the next chapter or Passino, 2005). 
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FIGURE 54.3 Fuzzy model reference learning control. 
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54.2.2 Expert and Planning Systems 


While fuzzy control techniques similar to those described above have been employed in a variety of indus- 
trial control applications, more general “knowledge-based controllers” have also been used successfully. 
For instance, there are “expert controllers” that are being used to directly control complex processes 
or in a supervisory role similar to that shown in Figure 54.2 (Antsaklis and Passino, 1993; Passino and 
Lunardhi, 1995). Others are being used to supervise conventional control algorithms. For instance, the 
work in Astrom et al. (1986) describes the use of expert supervisory systems for conventional adaptive 
controllers. Expert systems are also being used as the basis for learning controllers. 

In addition, there are planning systems (computer programs that emulate the way that experts plan) 
that have been used in path planning and high-level decisions about control tasks for robots (Antsaklis 
and Passino, 1993; Valavanis and Saridis, 1992; Dean and Wellman, 1991). A generic planning system, 
configured in the architecture of a standard control system, is shown in Figure 54.4. Here, the “problem 
domain” is the environment that the planner operates in, that is, the plant. There are measured outputs y; 
at step i (variables of the problem domain that can be sensed in real-time), control actions u; (the ways in 
which we can affect the problem domain), disturbances d; (that represent random events that can affect 
the problem domain and hence, the measured variable y;), and goals g; (what we would like to achieve in 
the problem domain). There are closed-loop specifications that quantify performance specifications and 
stability requirements. 

It is the task of the planner, shown in Figure 54.4, to monitor the measured outputs and goals and 
generate control actions that will counteract the effects of the disturbances and result in the goals and 
the closed-loop specifications to be achieved. To do this, the planner performs “plan generation” where 
it projects into the future (a finite number of steps using a model of the problem domain) and tries to 
determine a set of candidate plans. Next, this set of plans is pruned to one plan that is the best one to 
apply at the current time. The plan is executed and during execution the performance resulting from 
the plan is monitored and evaluated. Often, due to disturbances, plans will fail and hence the planner 
must generate a new set of candidate plans, select one, and then execute that one. While not pictured 
in Figure 54.4, some planning systems use “situation assessment” to try to estimate the state of the 
problem domain (this can be useful in execution monitoring and in plan generation), others perform 
“world modeling” where a model of the problem domain is developed in an online fashion (similar to 
online system identification), and “planner design” where information from the world modeler is used 
to tune the planner (so that it makes the right plans for the current problem domain). The reader will, 
perhaps, think of such a planning system as a general “self-tuning regulator.” For more details on the 
use of planning systems for control see Antsaklis and Passino (1993), Dean and Wellman (1991), and 
Passino (2005). 
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FIGURE 54.4 Closed-loop planning systems. 
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FIGURE 54.5 Neural predictive control. 


54.2.3 Neural Networks for Control 


There has been significant activity in the use of artificial neural networks for control (Hunt et al., 1994; 
Spooner et al., 2002; Farrell and Polycarpou, 2006). In this approach, engineers are trying to emulate 
the low-level biological functions of the brain and use these to solve challenging control problems. For 
instance, for some control problems we may train an artificial neural network to remember how to 
regulate a system by repeatedly providing it with examples of how to perform such a task. After the neural 
network has learned the task, it can be used to recall the control input for each value of the sensed output. 
Some other approaches to neural control, taken from Hunt et al. (1994 and the references therein), include 
a neural “internal model control” method and a “model reference structure” that is based on an approach 
to using neural networks for system identification. 

Still other neural control approaches bear some similarities to the FMRLC in Figure 54.3 in the sense 
that they automatically learn how to control a system by observing the behavior from that system. For 
instance, in Figure 54.5 we show a “neural predictive control” approach from Hunt et al. (1994) where 
one neural network is used as an identifier (structure) for the plant and another is used as a feedback 
controller for the plant that is tuned online. This tuning proceeds at each step by having the “optimizer” 
specify an input u’ for the neural model of the plant over some time interval. The predicted behavior of 
the plant y’ is obtained and used by the optimizer, along with y,, to pick the best parameters of the neural 
controller so that the difference between the plant and reference model outputs is as small as possible (if 
y’ predicts y well we would expect that the optimizer would be quite successful at tuning the controller). 
For more details on the multitude of techniques for using neural networks for control see Hunt et al. 
(1994). 


54.2.4 Genetic Algorithms for Control 


A genetic algorithm uses the principles of evolution, natural selection, and genetics from natural biological 
systems in a computer algorithm to simulate evolution (Goldberg, 1989). Essentially, the genetic algorithm 
performs a parallel, stochastic, but directed search to evolve the population that is most fit. It has been 
shown that a genetic algorithm can be used effectively in the (off-line) computer-aided-design of control 
systems because it can artificially “evolve” an appropriate controller that meets the performance specifi- 
cations to the greatest extent possible. To do this, the genetic algorithm maintains a population of strings 
where each represents a different controller and it uses the genetic operators of “reproduction” (which 
represents the “survival of the fittest” characteristic of evolution), “crossover” (which represents “mating”), 
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FIGURE 54.6 Genetic model reference adaptive controller. 


and “mutation” (which represents the random introduction of new “genetic material”), coupled with a 
“fitness measure” (which often quantifies the performance objectives) to generate successive generations 
of the population. After many generations, the genetic algorithm often produces an adequate solution to 
a control design problem since the stochastic, but directed, search helps avoid locally optimal designs and 
seeks to obtain the best design possible. 

Another more challenging problem is that of how to evolve controllers while the system is oper- 
ating, rather than in off-line design. Progress in this direction has been made by the introduction of 
the genetic model reference adaptive controller (GMRAC) shown in Figure 54.6 (Porter and Passino, 
1994). As in the FMRLC, the GMRAC uses a reference model to characterize the desired performance. 
For the GMRAC a genetic algorithm maintains a population of strings that represent candidate con- 
trollers. This genetic algorithm uses a process model (e.g., a linear model of the process) and data from 
the process to evaluate the fitness of each controller in the population at each time step. Using this 
fitness evaluation the genetic algorithm propagates controllers into the next generation via the stan- 
dard genetic operators. The controller that is the most fit one in the population is used to control the 
system. This allows the GMRAC to automatically evolve a controller from generation to generation 
(i.e., from one time step to the next) and hence to tune a controller in response to changes in the 
process (e.g., due to temperature variations, parameter drift, etc.) or due to an online change of the 
specifications in the reference model. Early indications are that the GMRAC seems quite promising 
as a new technique for stochastic adaptive control since it provides a unique feature where alternative 
controllers can be quickly applied to the problem if they appear useful, and because it has some inher- 
ent capabilities to learn via evolution of its population of controllers. There is, however, a significant 
amount of comparative and nonlinear analysis that needs to be done to fully evaluate this approach 
to control. 


54.3 Autonomous Control 


The goal of the field of autonomous control is to design control systems that automate enough functions 
so that they can independently perform well under significant uncertainties for extended periods of time 
even if there are significant system failures or disturbances. Below, we overview some of the basic ideas 
from Antsaklis and Passino (1993) and Passino (2005) on how to specify controllers that can in fact 
achieve high levels of autonomy. 
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54.3.1 The Intelligent Autonomous Controller 


Figure 54.7 shows a functional architecture for an intelligent autonomous controller with an interface 
to the process (plant) involving sensing (e.g., via conventional sensing technology, vision, touch, smell, 
etc.), actuation (e.g., via hydraulics, robotics, motors, etc.), and an interface to humans (e.g., a driver, 
pilot, crew, etc.) and other systems. The “execution level” has low-level numeric signal processing and 
control algorithms (e.g., PID, optimal, or adaptive control; parameter estimators, failure detection and 
identification (FDI) algorithms). The “coordination level” provides for tuning, scheduling, supervision, 
and redesign of the execution level algorithms, crisis management, planning and learning capabilities for 
the coordination of execution level tasks, and higher-level symbolic decision making for FDI and control 
algorithm management. The “management level” provides for the supervision of lower level functions 
and for managing the interface to the human(s). In particular, the management level will interact with the 
users in generating goals for the controller and in assessing capabilities of the system. The management 
level also monitors performance of the lower level systems, plans activities at the highest level (and 
in cooperation with the human), and performs high level learning about the user and the lower level 
algorithms. Applications that have used this type of architecture can be found in Antsaklis and Passino 
(1993), Valavanis and Saridis (1992), and Gazi et al. (2001). 

Intelligent systems/controllers (fuzzy, neural, genetic, expert, etc.) can be employed as appropriate in 
the implementation of various functions at the three levels of the intelligent autonomous controller (adap- 
tive fuzzy control may be used at the execution level, planning systems may be used at the management 
level for sequencing operations, and genetic algorithms may be used in the coordination level to pick 
an optimal coordination strategy). Hierarchical controllers composed of a hybrid mix of intelligent and 
conventional systems are commonly used in the intelligent control of complex dynamical systems. This 
is due to the fact that to achieve high levels of autonomy, we often need high levels of intelligence, which 
calls for incorporation of a diversity of decision-making approaches for complex dynamic reasoning. 

There are several fundamental characteristics that have been identified for intelligent autonomous 
control systems (see Valavanis and Saridis, 1992; Antsaklis and Passino, 1993, and the references therein). 
For example, there is generally a successive delegation of duties from the higher to lower levels and 
the number of distinct tasks typically increases as we go down the hierarchy. Higher levels are often 
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FIGURE 54.7 Intelligent autonomous controller. 
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concerned with slower aspects of the system’s behavior and with its larger portions, or broader aspects. 
There is then a smaller contextual horizon at lower levels, that is, the control decisions are made by 
considering less information. Higher levels are typically concerned with longer time horizons than lower 
levels. It is said that there is “increasing intelligence with decreasing precision as one moves from the lower 
to the higher levels” (see Valavanis and Saridis, 1992, and the references therein). At the higher levels 
there is typically a decrease in time scale density, a decrease in bandwidth or system rate, and a decrease 
in the decision (control action) rate. In addition, there is typically a decrease in granularity of models 
used, or equivalently, an increase in model abstractness at the higher levels. Finally, we note that there 
is an ongoing evolution of the intelligent functions of an autonomous controller so that by the time one 
implements its functions they no longer appear intelligent—just algorithmic. It is this evolution principle 
and the fact that implemented intelligent controllers are nonlinear controllers that many researchers 
feel more comfortable focusing on achieving autonomy rather than whether the resulting controller is 
intelligent. 


54.3.2 The Control-Theoretic View of Autonomy 


Next, it is explained how to incorporate the notion of autonomy into the conventional manner of thinking 
about control problems. Consider the general control system shown in Figure 54.8 where P is a model of 
the plant, C represents the controller, and T represents specifications on how we would like the closed- 
loop system to behave (i.e., closed-loop specifications). For some classical control problems the scope is 
limited so that C and P are linear and T simply represents, for example, stability, rise time, overshoot, and 
steady-state tracking error specifications. In this case, intelligent control techniques may not be needed. 
As engineers, the simplest solution that works is the best one. We tend to need more complex controllers 
for more complex plants (where, for example, there is a significant amount of uncertainty) and more 
demanding closed-loop specifications T (see Valavanis and Saridis, 1992; Antsaklis and Passino, 1993, 
and the references therein). 

Consider the case where 


1. P is so complex that it is most convenient to represent it with ordinary differential equations and 
discrete event system (DES) models (or some other “hybrid” mix of models) and for some parts of 
the plant the model is not known (i.e., it may be too expensive to find). 

2. T is used to characterize the desire to make the system perform well and act with high degrees 
of autonomy (i.e., so that the system performs well under significant uncertainties in the system 
and its environment for extended periods of time, and compensates for significant system failures 
without external intervention (Antsaklis and Passino, 1993). 


The general control problem is how to construct C, given P, so that T holds. The intelligent autonomous 
controller described briefly in the previous section provides a general architecture for C to achieve highly 
autonomous behavior specified by T for very complex plants P. 


FIGURE 54.8 Control system. 
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54.3.3 Modeling and Analysis* 


Conventional approaches to modeling P include the use of ordinary differential or difference equations, 
partial differential equations, stochastic models, models for hierarchical and distributed systems, and so 
on. However, often some portion of P is more easily represented with an automata model, Petri net, or 
some other DES model. Moreover, for analysis of the closed-loop system where there are a variety of 
intelligent and conventional controllers that are working together to form C (e.g., a planning system and 
a conventional adaptive controller) there is the need for “hybrid” modeling formalisms that can represent 
dynamics with differential equations and DES models. Control engineers use a model of the plant P to aid 
in the construction of the model of the controller C and this model is then implemented to control the real 
plant. In addition, models of C and P are used as formalisms to represent the dynamics of the closed-loop 
system so that analysis of the properties of the feedback system is possible before implementation (for 
redesign, verification, certification, and safety analysis). If the model of P is chosen to be too complex 
and C is very complex it will be difficult to develop and utilize mathematical approaches for the analysis 
of the resulting closed-loop system. Often we want the simplest possible model P that will allow for the 
development of the (simplest) controller C, and allow for it to be proven/demonstrated so that the closed- 
loop specifications T are met (of course, a separate more complex model P may be needed for simulation). 
Unfortunately, there is no clear answer to the question of how much or what type of modeling is needed 
for the plant P and there is no standardization of models for intelligent control in the way that there is 
for many areas of conventional control. Hence, although it is not exactly clear how to proceed with the 
modeling task, it is clear that knowledge of many different types of models may be needed, depending on 
the task at hand. 

Given the model of the plant P and the model of the controller C, the next task often considered by 
a control engineer is the use of analysis to more fully understand the behavior of P or the closed-loop 
system, and to show that when C and P are connected, the closed-loop specifications T are satisfied. Formal 
mathematical analysis can be used to verify stability, controllability, and observability properties. There 
is, in fact, a growing amount of literature on nonlinear analysis (e.g., stability and describing function 
analysis) of fuzzy control systems (both direct and adaptive [Jenkins and Passino, 1999; Spooner et al., 
2002; Farrell and Polycarpou, 2006]). There is a significant amount of activity in the area of nonlinear 
analysis of neural control systems and results in the past on nonlinear analysis of (numerical) learning 
control systems (see the later chapter in this section of this book and Hunt et al. (1994). There have already 
been some applications of DES theory to artificial intelligent (AI) planning systems and there have been 
recent results on stability analysis of expert control systems (Passino and Lunardhi, 1995). There has 
been some progress in defining models and developing approaches to analysis for some hybrid systems, 
but there is the need for much more work in this area. Many fundamental modeling and representation 
issues need to be reconsidered, different design objectives and control structures need to be examined, 
our repertoire of approaches to analysis and design needs to be expanded, and there is the need for more 
work in the area of simulation and experimental evaluation for hybrid systems. The importance of the 
solution to the hybrid control system analysis problem is based on the importance of solving the general 
control problem described above; that is, hybrid system analysis techniques could provide an approach to 
verifying the operation of intelligent controllers that seek to obtain truly autonomous operation. Finally, 
it must be emphasized that while formal verification of the properties of a control system is important, 
simulation and experimental evaluation always plays an especially important role also. 


54.4 Concluding Remarks 


We have provided a brief overview of the main techniques in the field of intelligent control and have pro- 
vided references for the reader who is interested in investigating the details of any one of these techniques. 


* The reader can find references for the work on modeling and analysis of intelligent control systems discussed in this 


section in [Passino, 2005]. 
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The intent of this chapter was to provide the reader with an overview of a relatively new area of control, 
while the intent of the next two chapters is to provide an introduction to two of the more well-developed 
areas in intelligent control—fuzzy and neural control. In a chapter so brief it seems important to indicate 
what has been omitted. We have not discussed: (1) FDI methods that are essential for a truly autonomous 
controller, (2) reconfigurable (fault tolerant) control strategies that use conventional nonlinear robust 
control techniques and intelligent control techniques, (3) sensor fusion and integration techniques that 
will be needed for autonomous control, (4) architectures for intelligent and autonomous control systems 
(e.g., alternative ways to structure interconnections of intelligent subsystems), (5) distributed intelligent 
systems (e.g., multiagent systems), (6) attentional systems, and (7) applications. 


54.5 Defining Terms 


Expert system: A computer program designed to emulate the actions of a human who is proficient 
at some task. Often the expert system is broken into two components: a “knowledge-base” 
that holds information about the problem domain, and an inference mechanism (engine) that 
evaluates the current knowledge and decides what actions to take. An “expert controller” is an 
expert system that is designed to automate the actions of a human operator who controls a 
system. 

Fuzzy systems: A type of knowledge-based system that uses fuzzy logic for knowledge representation and 
inference. It is composed of four primary components: the fuzzification interface, the rule-base 
(a knowledge-base that is composed of rules), an inference mechanism, and a defuzzification 
interface. A fuzzy system that is used to control a system is called a “fuzzy controller.” 

Planning system: Computer program designed to emulate human planning activities. These may be a 
type of expert system that has a special knowledge-base that has plan fragments and strategies 
for planning and an inference process that generates and evaluates alternative plans. 

Neural network: Artificial hardware (e.g., electrical circuits) designed to emulate biological neural 
networks. These may be simulated on conventional computers or on specially designed “neural 
processors.” 

Genetic algorithm: A genetic algorithm uses the principles of evolution, natural selection, and genetics 
from natural biological systems in a computer algorithm to simulate evolution. Essentially, 
the genetic algorithm performs a parallel, stochastic, but directed search to evolve the most fit 
population. 

Intelligent autonomous control system: A control system that uses conventional and intelligent control 
techniques to provide enough automation so that the system can independently perform well 
under significant uncertainties for extended periods of time even if there are significant system 
failures or disturbances. 
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For Further Information 


While the books and articles referenced above (particularly, Passino, 2005) should provide the reader 
with an introduction to the area of intelligent control, there are other sources that may also be useful. For 
instance, there are many relevant conferences including: (1) IEEE International Symposium on Intelligent 
Control, (2) American Control Conference, (3) IEEE Conference on Decision and Control, (4) IEEE Confer- 
ence on Control Applications, and (5) IEEE International Conference on Systems, Man, and Cybernetics. 
In addition, there are many conferences on fuzzy systems, expert systems, genetic algorithms, and neural 
networks where applications to control are often studied. There are many journals that cover the topic 
of intelligent control including: (1) IEEE Control Systems Magazine, (2) IEEE Trans. on Control Systems 
Technology, (3) IEEE Trans. on Systems, Man, and Cybernetics, (4) IEEE Trans. on Fuzzy Systems, (5) IEEE 
Trans. on Neural Networks, (6) Engineering Applications of Artificial Intelligence, (7) Journal of Intelligent 
and Robotic Systems, (8) Applied Artificial Intelligence, and (9) Journal of Intelligent and Fuzzy Systems. 
There are many other journals on expert systems, neural networks, genetic algorithms, and fuzzy systems 
where applications to control can often be found. The professional societies most active in intelligent 
control are the IEEE Control Systems Society, International Federation on Automatic Control, and the 
IEEE Systems, Man, and Cybernetics Society. 
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55.1 Introduction 


When confronted with a control problem for a complicated physical process, the control engineer usually 
follows a predetermined design procedure, which begins with the need for understanding the process and 
the primary control objectives. A good example of such a process is that of an automobile “cruise control,” 
designed with the objective of providing the automobile with the capability of regulating its own speed 
at a driver-specified set-point (e.g., 55 mph). One solution to the automotive cruise control problem 
involves adding an electronic controller that can sense the speed of the vehicle via the speedometer 
and actuate the throttle position so as to regulate the vehicle speed at the driver-specified value even if 
there are road grade changes, head-winds, or variations in the number of passengers in the automobile. 
Control engineers typically solve the cruise control problem by (1) developing a model of the automobile 
dynamics (which may model vehicle and power train dynamics, road grade variations, etc.), (2) using 
the mathematical model to design a controller (e.g., via a linear model develop a linear controller with 
techniques from classical control), (3) using the mathematical model of the closed-loop system and 
mathematical or simulation-based analysis to study its performance (possibly leading to redesign), and 
(4) implementing the controller via, for example, a microprocessor, and evaluating the performance of 
the closed-loop system (again possibly leading to redesign). 
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The difficult task of modeling and simulating complex real-world systems for control systems devel- 
opment, especially when implementation issues are considered, is well documented. Even if a relatively 
accurate model of a dynamical system can be developed, it is often too complex to use in controller devel- 
opment, especially for many conventional control design procedures that require restrictive assumptions 
for the plant (e.g., linearity). It is for this reason that in practice conventional controllers are often 
developed via simple crude models of the plant behavior that satisfy the necessary assumptions, and via 
the ad hoc tuning of relatively simple linear or nonlinear controllers. Regardless, it is well understood 
(although sometimes forgotten) that heuristics enter the design process when the conventional control 
design process is used as long as one is concerned with the actual implementation of the control system. It 
must be acknowledged, however, that conventional control engineering approaches that use appropriate 
heuristics to tune the design have been relatively successful (the vast majority of all controllers currently 
in operation are conventional PID controllers). One may ask the following questions: How much of the 
success can be attributed to the use of the mathematical model and conventional control design approach, 
and how much should be attributed to the clever heuristic tuning that the control engineer uses upon 
implementation? If we exploit the use of heuristic information throughout the entire design process can 
we obtain higher performance control systems? 

Fuzzy control provides a formal methodology for representing, manipulating, and implementing a 
human’s heuristic knowledge about how to control a system. Fuzzy controller design involves incor- 
porating human expertise on how to control a system into a set of rules (a rule-base). The inference 
mechanism in the fuzzy controller reasons over the information in the knowledge-base, the process out- 
puts, and the user-specified goals to decide what inputs to generate for the process so that the closed-loop 
fuzzy control system will behave properly (e.g., so that the user-specified goals are met). From the cruise 
control example discussed above, it is clear that anyone who has experience in driving a car can practice 
regulating the speed about a desired set-point and load this information into a rule-base. For instance, 
one rule that a human driver may use is “IF speed is lower than the set-point THEN press down further on 
the accelerator pedal.” A rule that would represent even more detailed information about how to regulate 
the speed would be “IF speed is lower than the set-point AND speed is approaching the set-point very fast 
THEN release the accelerator pedal by a small amount.” This second rule characterizes our knowledge 
about how to make sure that we do not overshoot our desired (goal) speed. Generally speaking, if we 
load very detailed expertise into the rule-base we enhance our chances of obtaining better performance. 
Overall, the focus in fuzzy control is on the use of heuristic knowledge to achieve good control, whereas 
in conventional control the focus is on the use of a mathematical model for control systems development 
and subsequent use of heuristics in implementation. 


55.1.1 Philosophy of Fuzzy Control 


Due to the substantial amount of hype and excitement about fuzzy control, it is important to begin by 
providing a sound control engineering philosophy for this approach. First, there is a need for the control 
engineer to assess what (if any) advantages fuzzy control methods have over conventional methods. Time 
permitting, this must be done by careful comparative analyses involving modeling, mathematical analysis, 
simulation, implementation, and a full engineering cost-benefit analysis (which involves issues of cost, 
reliability, maintainability, flexibility, lead-time to production, etc.). When making the assessment of 
what control technique to use, the engineer should be cautioned that most work in fuzzy control to date 
has only focused on its advantages and has not taken a critical look at what possible disadvantages there 
could be to using it. For example, the following questions are cause for concern: 


¢ Will the behaviors observed by a human expert include all situations that can occur due to distur- 
bances, noise, or plant parameter variations? 

¢ Can the human expert realistically and reliably foresee problems that could arise from closed-loop 
system instabilities or limit cycles? 
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¢ Will the expert be able to effectively incorporate stability criteria and performance objectives (e.g., 
rise-time, overshoot, and tracking specifications) into a rule-base to ensure that reliable operation 
can be obtained? 

« Can an effective and widely used synthesis procedure be devoid of mathematical modeling and 
subsequent use of proven mathematical analysis tools? 


These questions may seem even more troublesome if: (1) the control problem involves a “critical envi- 
ronment” where the failure of the control system to meet performance objectives could lead to loss of 
human life or an environmental disaster (e.g., in aircraft or nuclear power plant control), or (2) if the 
human expert’s knowledge implemented in the fuzzy controller is somewhat inferior to that of a very 
experienced specialist that we expect to have design the control system (different designers have different 
levels of expertise). Clearly, then, for some applications there is a need for a methodology to develop, 
implement, and evaluate fuzzy controllers to ensure that they are reliable in meeting their performance 
specifications. 

As it is discussed above, the standard control engineering methodology involves repeatedly coordi- 
nating the use of modeling, controller (re)design, simulation, mathematical analysis, and experimen- 
tal evaluations to develop control systems. What is the relevance of this established methodology to 
the development of fuzzy control systems? Engineering a fuzzy control system uses many ideas from 
the standard control engineering methodology, except that in fuzzy control it is often said that a for- 
mal mathematical model is assumed unavailable so that mathematical analysis is impossible. While it 
is often the case that it is difficult, impossible, or cost-prohibitive to develop an accurate mathemat- 
ical model for many processes, it is almost always possible for the control engineer to specify some 
type of approximate model of the process (after all, we do know what physical object we are trying to 
control). Indeed, it has been our experience that most often the control engineer developing a fuzzy 
control system does have a mathematical model available. While it may not be used directly in con- 
troller design, it is often used in simulation to evaluate the performance of the fuzzy controller before 
it is implemented (and it is often used for rule-base redesign). Certainly there are some applications 
where one can design a fuzzy controller and evaluate its performance directly via an implementation. In 
such applications one may not be overly concerned with a high performance level of the control system 
(e.g., for some commercial products such as washing machines or a shaver). In such cases, there may 
thus be no need for conducting simulation-based evaluations (requiring a mathematical model) before 
implementation. In other applications there is the need for a high level of confidence in the reliabil- 
ity of the fuzzy control system before it is implemented (e.g., in systems where there is a concern for 
safety). 

In addition to simulation-based studies, one approach to enhancing our confidence in the reliability 
of fuzzy control systems is to use the mathematical model of the plant and nonlinear analysis for (1) 
verification of stability and performance specifications and (2) possible redesign of the fuzzy controller 
(for an overview of the results in this area see [1]). Some may be confident that a true expert would 
never need anything more than intuitive knowledge for rule-base design, and therefore, never design a 
faulty fuzzy controller. However, a true expert will certainly use all available information to ensure the 
reliable operation of a control system including approximate mathematical models, simulation, nonlin- 
ear analysis, and experimentation. We emphasize that mathematical analysis cannot alone provide the 
definitive answers about the reliability of the fuzzy control system because such analysis proves properties 
about the model of the process, not the actual physical process. It can be argued that a mathematical 
model is never a perfect representation of a physical process; hence, while nonlinear analysis (e.g., of 
stability) may appear to provide definitive statements about control system reliability, it is understood 
that such statements are only accurate to the extent that the mathematical model is accurate. Nonlinear 
analysis does not replace the use of common sense and evaluation via simulations and experimentation; 
it simply assists in providing a rigorous engineering evaluation of a fuzzy control system before it is 
implemented. 
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It is important to note that the advantages of fuzzy control often become most apparent for very 
complex problems where we have an intuitive idea about how to achieve high performance control. 
In such control applications an accurate mathematical model is so complex (i.e., high order, nonlin- 
ear, stochastic, with many inputs and outputs) that it is sometimes not very useful for the analysis 
and design of conventional control systems (since assumptions needed to utilize conventional control 
design approaches are often violated). The conventional control engineering approach to this prob- 
lem is to use an approximate mathematical model that is accurate enough to characterize the essential 
plant behavior, yet simple enough so that the necessary assumptions to apply the analysis and design 
techniques are satisfied. However, due to the inaccuracy of the model, upon implementation the devel- 
oped controllers often need to be tuned via the “expertise” of the control engineer. The fuzzy control 
approach, where explicit characterization and utilization of control expertise is used earlier in the design 
process, largely avoids the problems with model complexity that are related to design. That is, for the 
most part fuzzy control system design does not depend on a mathematical model unless it is needed 
to perform simulations to gain insight into how to choose the rule-base and membership functions. 
However, the problems with model complexity that are related to analysis have not been solved (i.e., 
analysis of fuzzy control systems critically depends on the form of the mathematical model); hence, 
it is often difficult to apply nonlinear analysis techniques to the applications where the advantages of 
fuzzy control are most apparent. For instance, as shown in [1], existing results for stability analysis of 
fuzzy control systems typically require that the plant model be deterministic, satisfy some continuity 
constraints, and sometimes require the plant to be linear or “linear-analytic.” The only results for analysis 
of steady-state tracking error of fuzzy control systems, and the existing results on the use of describ- 
ing functions for analysis of limit cycles, essentially require a linear time-invariant plant (or one that 
has a special form so that the nonlinearities can be bundled into a separate nonlinear component in 
the loop). 

The current status of the field, as characterized by these limitations, coupled with the importance 
of nonlinear analysis of fuzzy control systems, make it an open area for investigation that will help 
establish the necessary foundations for a bridge between the communities of fuzzy control and non- 
linear analysis. Clearly fuzzy control technology is leading the theory; the practitioner will proceed 
with the design and implementation of many fuzzy control systems without the aid of nonlinear 
analysis. In the mean time, theorists will attempt to develop a mathematical theory for the verifica- 
tion and certification of fuzzy control systems. This theory will have a synergistic effect by driving 
the development of fuzzy control systems for applications where there is a need for highly reliable 
implementations. 


55.1.2 Summary 


The focus of this chapter is on providing a practical introduction to fuzzy control (a “users guide”) in 
the style of the book [2] (which is available at Kevin M Passino’s web site for a free download); hence, 
we omit discussions of mathematical analysis of fuzzy control systems and invite the interested reader to 
investigate this topic further by consulting the bibliographic references. The remainder of this chapter 
is arranged as follows. We begin by providing a general mathematical introduction to fuzzy systems in 
a tutorial fashion. Next, we introduce a rotational inverted pendulum “theme problem.” Many details 
on control design using principles of fuzzy logic are presented via this theme problem. We perform 
comparative analyses for fixed (nonadaptive) fuzzy and linear controllers. Following this we introduce 
the area of adaptive fuzzy control and show how one adaptive fuzzy technique has proven to be particularly 
effective for balancing control of the inverted pendulum. In the concluding remarks we explain how the 
area of fuzzy control is related to other areas in the field of intelligent control and what research needs to 
be performed as the field of fuzzy control matures. 
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55.2 Introduction to Fuzzy Control 


The functional architecture of the fuzzy system (controller)* is composed ofa rule-base (containing a fuzzy 
logic quantification of the expert’s linguistic description of how to achieve good control), an inference 
mechanism (which emulates the expert’s decision-making in interpreting and applying knowledge about 
how to do good control), a fuzzification interface (which converts controller inputs into information that 
the inference mechanism can easily use to activate and apply rules), and a defuzzification interface (which 
converts the conclusions of the inference mechanism into actual inputs for the process). Here we describe 
each of these four components in more detail (see Section 55.4 for a block diagram) [3-5]. 


55.2.1 Linguistic Rules 


For our purposes, a fuzzy system is a static nonlinear mapping between its inputs and outputs (i.e., it is 
not a dynamical system). It is assumed that the fuzzy system has inputs u; € U; where i= 1,2,...,n and 
outputs y; € Yj where i= 1,2,...,m. The ordinary (“crisp”) sets U4; and )j are called the “universes of 
discourse” for u; and y;, respectively (in other words they are their domains). 

To specify rules for the rule-base the expert will use a “linguistic description”; hence, linguistic expres- 
sions are needed for the inputs and outputs and the characteristics of the inputs and outputs. We will 
use “linguistic variables” (constant symbolic descriptions of what are in general time-varying quantities) 
to describe fuzzy system inputs and outputs. For our fuzzy system, linguistic variables denoted by u; are 
used to describe the inputs u;. Similarly, linguistic variables denoted by y; are used to describe outputs y;. 
For instance, an input to the fuzzy system may be described as 1; =“velocity error” and an output from 
the fuzzy system may be y; =“voltage in.” 

Just as u; and y; take on values over each universe of discourse U4; and Jj, respectively, linguistic 
variables u; and y; take on “linguistic values” that are used to describe characteristics of the variables. 
Let Al denote the jth linguistic value of the linguistic variable u; defined over the universe of discourse 
U;. If we assume that there exist many linguistic values defined over U/;, then the linguistic variable 4; 
takes on the elements from the set of linguistic values denoted by A; = {A :j =1,2,..., Nj} (sometimes 
for convenience we will let the j indices take on negative integer values). Similarly, let BI denote the jth 
linguistic value of the linguistic variable y; defined over the universe of discourse Jj. The linguistic variable 
7; takes on elements from the set of linguistic values denoted by B; = {BP :p=1,2,...,Mj} (sometimes 
for convenience we will let the p indices take on negative integer values). Linguistic values are generally 
zero, and “negative big” (i.e., adjectives). 

The mapping of the inputs to the outputs for a fuzzy system is in part characterized by a set of 
condition — action rules, or in modus ponens (If... Then) form, 


0 


expressed by descriptive terms such as “positive large, 


If (antecedent) Then (consequent). (55.1) 


As usual, the inputs of the fuzzy system are associated with the antecedent, and the outputs are associated 
with the consequent. These If... Then rules can be represented in many forms. Two standard forms, multi- 
input multi-output (MIMO) and multi-input single output (MISO) are considered here. The MISO form 
of a linguistic rule is: 


If i is AL and ii is Ak and,..., and i, is Al Then jy, is By (55.2) 


It is a whole set of linguistic rules of this form that the expert specifies on how to control the system. 


Note that if i;=“velocity error” and A’ =“positive large,” then “it; is A’,” a single term in the antecedent 
of the rule, means “velocity error is positive large.” It can be easily shown that the MIMO form for a rule 


* Sometimes a fuzzy controller is called a “fuzzy logic controller” or even a “fuzzy linguistic controller” since, as we will see, 


it uses fuzzy logic in the quantification of linguistic descriptions. 
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(i.e., one with consequents that have terms associated with each of the fuzzy controller outputs) can be 
decomposed into a number of MISO rules (using simple rules from logic). We assume that there are a 
total of R rules in the rule-base numbered 1, 2,..., R. For simplicity we will use tuples (j, k, ..., 1; p, q); to 
denote the ith MISO rule of the form given in Equation 55.2. Any of the terms associated with any of the 
inputs for any MISO rule can be included or omitted. Finally, we naturally assume that the rules in the 
rule-base are distinct (i.e., there are no two rules with exactly the same antecedents and consequents). 


55.2.2 Fuzzy Sets, Fuzzy Logic, and the Rule-Base 


Fuzzy sets and fuzzy logic are used to heuristically quantify the meaning of linguistic variables, linguistic 
values, and linguistic rules that are specified by the expert. The concept of a fuzzy set is introduced by first 
defining a “membership function.” Let U/; denote a universe of discourse and A’ € A; denote a specific 
linguistic value for the linguistic variable u;. The function 1(u;) associated with At that maps Uj; to [0, 1] 
is called a “membership function.” This membership function describes the “certainty” that an element 
of U;, denoted uj;, with a linguistic description u4;, may be classified linguistically as Al, Membership 
functions are generally subjectively specified in an ad hoc (heuristic) manner from experience or intuition. 
For instance, if 4; = [—150, 150], uj=“velocity error,” and A\=“positive large,” then |1(u;) may be a bell- 
shaped curve that peaks at one at uj = 75 and is near zero when uj < 50 or uj; > 100. Then if uj = 75, 
(75) = 1 so that we are absolutely certain that u; is “positive large.” If uj = —25 then j(—25) is very near 
zero, which represents that we are very certain that u; is not “positive large.” Clearly, many other choices 
for the shape of the membership function are possible (e.g., triangular and trapezoidal shapes) and these 
will each provide a different meaning for the linquistics that they quantify. Below, we will show how to 
specify membership functions for a fuzzy controller for the rotational inverted pendulum. 

Given a linguistic variable 4; with a linguistic value Al defined on the universe of discourse U/;, and 


membership function t ,j(ui) (membership function associated with the fuzzy set Al) that maps U/; to 


[0, 1], a “fuzzy set” denoted with Al is defined as 
Ay = {(uis Bg (ui)) : ui € Ub}. (55.3) 


Next, we specify some set-theoretic and logical operations on fuzzy sets. Given fuzzy sets A} and A? 
associated with the universe of discourse U4; (Nj = 2), with membership functions denoted Al (uj) and 
Wt 42(uj), respectively, A} is defined to be a “fuzzy subset” of A}, denoted by A} ‘e A}, if yi (ui) < Wy42(ui) 
for all uj € Uj. 


The intersection of fuzzy sets Aj and A? which are defined on the universe of discourse Uj; is a fuzzy 
set, denoted by Aj 9 A?, with a membership function defined by either: 


Minimum: |yiqg2 = min{W 41 (uj), W42(ui) : ui € Uj}, 
i i i i (55.4) 
Algebraic Product: ying? = {hai (ui) 42 (ui) : uj; € Ui}. 


Suppose that we use the notation x * y = min{x, y} or at other times we will use it to denote the product 
x*y = xy (* is sometimes called the “triangular norm”). Then 1 41 (uj) * W42(uj) is a general represen- 
tation for the intersection of two fuzzy sets. In fuzzy logic, intersection is used to represent the “and” 
operation. 

The union of fuzzy sets A} and A?, which are defined on the universe of discourse Uj, is a fuzzy set 
denoted A} U A?, with a membership function defined by either: 


Maximum: = i 4iy,42(ui) = max{jr gi (ui), Wa2(ui) : ui € Ui}, oe 
+ 1 £ iz 55.5 
Algebraic Sum: | 41442(ui) = {he gi (ui) + Wye (ui) — begs (Mi) 42 (ui) ¢ ui € Ui}. 
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Suppose that we use the notation x @ y = max{x, y} or at other times we will use it to denote x@ 
y=x+y-—xy (@ is sometimes called the “triangular conorm”). Then al (ui) Bw az (ui) is a general 
representation for the union of two fuzzy sets. In fuzzy logic, union is used to represent the “or” operation. 

The intersection and union above are both defined for fuzzy sets that lie on the same universe of 
discourse. The fuzzy Cartesian product is used to quantify operations on many universes of discourse. 
If Al, AK ..., Al) are fuzzy sets defined on the universes of discourse U4,U2,...,Un, respectively, their 
Cartesian product is a fuzzy set (sometimes called a “fuzzy relation”), denoted by Al x Ak xx Al, 
with a membership function defined by 


Mab scabs (I> MDs +9 Mn) = Wg) (UL) # Hage (la) + ay (tn) (55.6) 


Next, we show how to quantify the linguistic elements in the antecedent and consequent of the linguistic 
If... Then rule with fuzzy sets. For example, suppose we are given the If... Then rule in MISO form in 
Equation 55.2. Define the fuzzy sets: 


Ay = ((uy phy (ua) m1 € Uh} 
AS = {(ua, gk (U2) : U2 € U2} 


(55.7) 


Al, = {(un, Il (Un)) Un € Un} 
By = {(Yqo pp (Yq) Yq € Yq}: 


These fuzzy sets quantify the terms in the antecedent and the consequent of the given If... Then rule, to 
make a “fuzzy implication” 


If A! and AS and,...,and At Then B’, (55.8) 


where the fuzzy sets Al, Ak isan Als and B, are defined in Equation 55.8. Therefore, the fuzzy set A, is 
associated with, and quantifies, the meaning of the linguistic statement “i; is A ” and BE quantifies the 
meaning of “yg is Be Each rule in the rule-base (j,k, ...,1;p,q)i,i= 1,2,...,R is represented with such 
a fuzzy implication (a fuzzy quantification of the linguistic rule). The reader who is interested in more 
mathematical details on fuzzy sets and fuzzy logic should consult [6]. 


55.2.3 Fuzzification 


Fuzzy sets are used to quantify the information in the rule-base, and the inference mechanism operates 
on fuzzy sets to produce fuzzy sets; hence, we must specify how the fuzzy system will convert its numeric 
inputs u; € U4; into fuzzy sets (a process called “fuzzification”) so that they can be used by the fuzzy system. 
Let U/* denote the set of all possible fuzzy sets that can be defined on U4;. Given u; € Uj, fuzzification 
transforms u; to a fuzzy set denoted by Ae defined* over the universe discourse U/;. This transformation 
is produced by the fuzzification operator F defined by F : Uj; > U5, where F(uj) := Am, Quite often 
“singleton fuzzification” is used, which produces a fuzzy set Ale €U;* with a membership function 
defined by 


1 x=4Uj, 


I ggoe(X) = | (55.9) 


0 otherwise 


(any fuzzy set with this form for its membership function is called a “singleton”). Singleton fuzzification 
is generally used in implementations since, without the presence of noise, we are absolutely certain 


* In this section, as we introduce various fuzzy sets. We will always use a hat over any fuzzy set whose membership function 


changes dynamically over time as uj changes. 
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that uj takes on its measured value (and no other value) and since it provides certain savings in the 
computations needed to implement a fuzzy system (relative to, e.g., “Gaussian fuzzification” which would 
involve forming bell-shaped membership functions about input points). Throughout the remainder of 
this chapter we use singleton fuzzification. 


55.2.4 The Inference Mechanism 


The inference mechanism has two basic tasks: (1) determining the extent to which each rule is relevant 
to the current situation as characterized by the inputs uj, i= 1,2,...,n (we call this task “matching”), 
and (2) drawing conclusions using the current inputs u; and the information in the rule-base (we call 


this task an “inference step”). For matching note that Al x Ak x--.x Al is the fuzzy set representing the 
antecedent of the ith rule (j,k, ...,1; p, q)i (there may be more than one such rule with this antecedent). 
Suppose that at some time we get inputs u;, i = 1,2,...,n, and fuzzification produces Af”, AN, ..., AM”, 


the fuzzy sets representing the inputs. The first step in matching involves finding fuzzy sets Al ; Ak, Sey 
with membership functions 


aj (uy) = j (UL) * 4 fuz (U 
gi ( 1) bi ( 1) HAs (u1) 


ej (U2) = He gk (U2) * Hj fe (U2) 


Meal (un) = Wal (un) * MA fur (un) 


(for all j,k,...,1) that combine the fuzzy sets from fuzzification with the fuzzy sets used in each of the 
terms in the antecedents of the rules. If singleton fuzzification is used then each of these fuzzy sets is 
a singleton that is scaled by the antecedent membership function (e.g., 4 4i (41) = bi (41) for w= uy 
1 1 
and Wai (u,) = 0 for u, # u,). Second, we form membership values j1j(u1, u2,..., Un) for each rule that 
1 


represent the overall certainty that rule i matches the current inputs. In particular, we first let 


Wii(uy, U2,...,Un) = Wj (m1) * gx (ua) * 7 War (Un) (55.10) 


be the membership function for At x Ak x ++. x Al. Notice that since we are using singleton fuzzification, 
we have 
Pitty, ta, .- Hn) = We gj (U1) # Wg (tea) # + Weal in) (55.11) 


for uj=uj, and [j(t1,u2,...,U,):=0 for uj Auj, i=1,2,...,n. Since the wu; are given, 
Wj (U1), W4k(U2),..-, Wai (Un) are constants. Define 
Ay 2 n 


Hi(U1, Ua, --> Un) = Wai (M1) * Wyk (2) # + * Wal (Un), (55.12) 


which is simply a function of the inputs uj. We use [j(u1, U2,...,U,) to represent the certainty that 
the antecedent of rule i matches the input information. This concludes the process of matching input 
information with the antecedents of the rules. 

Next, the inference step is taken by computing, for the ith rule (j,k, ..., 1; p, q)i, the “implied fuzzy set” 
Bi with membership function 


Hpi (Yq) = Hi(U1s U2, - Un) * Lge (Yq): (55.13) 


The implied fuzzy set Bi specifies the certainty level that the output should be a specific crisp output yg 
within the unverse of discourse yj, taking into consideration only rule i. Note that since |1;(uy, ua, -.-, Un) 
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will vary with time so will the shape of the membership functions 1; (y,) for each rule. Alternatively, 
q 


the inference mechanism could, in addition, compute the “overall implied fuzzy set” By with membership 
function 


1g, q) = Hpi (Vg) ® bpp (¥q) B® Mpeg) (55.14) 


that represents the conclusion reached considering all the rules in the rule-base at the same time (notice 
that determining By can, in general, require significant computational resources). 

Using the mathematical terminology of fuzzy sets, the computation of By, (yq) is said to be produced 
by a “sup-star compositional rule of inference.” The “sup” in this terminology corresponds to the © 
operation and the “star” corresponds to *«. “Zadeh’s compositional rule of inference” [7] is the special 
case of the sup-star compositional rule of inference when max is used for © and min is used for *. The 
overall justification for using the above operations to represent the inference step lies in the fact that we 
can be no more certain about our conclusions than we are about our premises (antecedents). The operations 
performed in taking an inference step adhere to this principle. To see this, study Equation 55.13 and note 
that the scaling from 1 ;(u1, u2,..., Un) that is produced by the antecedent matching process ensures that 
sup), {wu Bi (¥q)} < wilui, v2, ..., Un) (a similar statement holds for the overall implied fuzzy set). 

Up to this point we have used fuzzy logic to quantify the rules in the rule-base, fuzzification to produce 
fuzzy sets characterizing the inputs, and the inference mechanism to produce fuzzy sets representing the 
conclusions that it reaches considering the current inputs and the information in the rule-base. Next, we 
look at how to convert this fuzzy set quantification of the conclusions to a numeric value that can be input 
to the plant. 


55.2.5 Defuzzification 


A number of defuzzification strategies exist. Each provides a means to choose a single output (which we 
denote with ia) based on either the implied fuzzy sets or the overall implied fuzzy set (depending on 
the type of inference strategy chosen). First, we present typical defuzzification techniques for the overall 
implied fuzzy set By: 


sp 


¢ Max Criteria: A crisp output a is chosen as the point on the output universe of discourse ), for 


which the overall implied fuzzy set By achieves a maximum that is, 


yar E€ {» ap {ha, (| : (55.15) 
4 


Since the supremum can occur at more than one point in ), one also needs to specify a strategy on 


how to pick only one point for ye (e.g., choosing the smallest value). Often this defuzzification 
strategy is avoided due to this ambiguity. 


¢ Mean of Maximum: A crisp output Var 


is chosen to represent the mean value of all elements whose 
membership in B, is a maximum. We define bj" as the supremum of the membership function of 


By over the universe of discourse Yq. Moreover, we define a fuzzy set Be Evy with a membership 
function defined as 


Lp, (eq) =o", 
nye A 55.16 
MBs Oa) ; otherwise, : 
then a crisp output, using the mean of maximum method, is defined as 
sp 24° MB Va) Vg 
oS m a (55.17) 


ty, M5 a) “dyq 
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Note that the integrals in Equation 55.17 must be computed at each time instant since they depend 
on Bg which changes with time. This can require excessive computational resources; hence, this 
defuzzification technique is often avoided in practice. 

¢ Center of Area (COA): A crisp output yg is chosen as the COA for the membership function of 
the overall implied fuzzy set B,. For a continuous output universe of discourse ¥, the COA output 
is denoted by 

crisp _ Sy, Yq" bg, a) dyq 


y (55.18) 
: Sy, #3, a) 4 


Note that similar to the mean of the maximum method this defuzzification approach can be 
computationally expensive. Also, the fuzzy system must be defined so that Sy, MB, (yq)dyq A 0 for 
all u;. 


Next, we specify typical defuzzification techniques for the implied fuzzy sets Bi: 


¢ Centroid: A crisp output ieee is chosen using the centers of each of the output membership 


functions and the maximum certainty of each of the conclusions represented with the implied 
fuzzy sets and is given by 


ee sup,, {Mj (¥q)} 
Die UP, (hj, Og)! 


crisp 


ys (55.19) 


where Oe is the COA of the membership function of BE associated with the implied fuzzy set Bi 
for the ith rule (j,k,...,1;p,q)i. Notice that supy, (MB! (yq)} is often very easy to compute since if 
q 


[pe (Yq) = 1 for at least one y, (which is the normal way to define membership functions), then for 
4 
many inference strategies sup,, {HL pi Vq)} = Wilt, U2, ---, Un) which as already been computed in 
q 
the matching process. Notice that the fuzzy system must be defined so that ye 1 SUPy, {uw Bi (¥q)} #0 


for all u;. 
¢ Center of Gravity (COG): A crisp output Vt is chosen using the COA and area of each implied 
fuzzy set and is given by 


Diss oy Jy, Ha: Va) Aq 
ya Sy, Bi (Yq) 4yq 


crisp 


yq 


(55.20) 


where Ch is the COA of the membership function of B associated with the implied fuzzy set Bi 

for the ith rule (j,k,...,1;p,q)i. Notice that COG can be easy to compute since it is often easy to 

find closed-form expressions for Ly, lpi (¥q) dvq, which is the area under a membership function. 
q 


Notice that the fuzzy system must be defined so that yen 1 by, Mai (Vq) 4¥q A 0 for all uj. 
q 


Overall, we see that using the overall implied fuzzy set in defuzzification is often undesirable for two 
reasons: (1) the overall implied fuzzy set B, is itself difficult to compute in general, and (2) the defuzzifi- 
cation techniques based on an inference mechanism that provides By are also difficult to compute. It is 
for this reason that most existing fuzzy controllers (including the ones in this chapter) use defuzzification 
techniques based on the implied fuzzy sets such as Centroid or COG. 


55.3 Theme Problem: Rotational Inverted Pendulum 


One of the classic problems in the study of nonlinear systems is that of the inverted pendulum. The 
primary control problem one considers with such a system is regulating the position of the pendulum 
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(typically a rod with mass at the endpoint) to the vertical (up) position; that is, “balanced.” A secondary 
problem is that of “swinging up” the pendulum from its rest position (vertical down). Often, actuation 
is accomplished either via a motor at the base of the pendulum (at the hinge), or via a cart through 
translational motion. In this example actuation of the pendulum is accomplished through rotation of a 
separate, attached link, referred to, henceforth, as the “base.” 


55.3.1 Experimental Apparatus 


The test bed consists of three primary components: the plant, digital and analog interfaces, and the 
digital controller. The overall system is shown in Figure 55.1 where the three components can be clearly 
identified [8]. The plant is composed of a pendulum and a rotating base made of aluminum rods, two 
optical encoders as the angular position sensors with effective resolutions of 0.2 degrees for the pendulum 
and 0.1 degrees for the base, and a large, high-torque permanent-magnet DC motor (with rated stall 
torque of 5.15Nm). As the base rotates through the angle 9) the pendulum is free to rotate (high 
precision bearings are utilized) through its angle 9; made with the vertical. 

Interfaces between the digital controller and the plant consist of two data acquisition cards and some 
signal conditioning circuitry, structured for the two basic functions of sensor integration and control 
signal generation. The signal conditioning is accomplished via a combination of several logic gates to 
filter quadrature signals from the optical encoders, which are then processed through a separate data 
acquisition card to utilize the four 16-bit counters (accessed externally to count pulses from the circuitry 
itself). Another card supplies the control signal interface through its 12-bit D/A converter (to generate 
the actual control signal), while the board’s 16-bit timer is used as a sampling clock. The computer used 
for control is a personal computer with its Intel 80486DX processor operating at 50 MHz. The real-time 
codes for control are written in C. 


55.3.2 Mathematical Model 


For brevity, and because this system is a popular example for nonlinear control, we omit details of the 
necessary physics and geometry for modeling. The differential equations that approximately describe the 
dynamics of the plant are given by 


89 = —ap%o + Kp Va, (55.21) 

. Gis l re 

b= 7 —  sin(01) + K1605 (55.22) 
1 1 


where, again, 00 is the angular displacement of the rotating base, 99 is the angular speed of the rotating 
base, 6; is the angular displacement of the pendulum, 6, is the angular speed of the pendulum, vj is the 
motor armature voltage, Kp and ap are parameters of the DC motor with torque constant Kj, g is the 
acceleration due to gravity, m; is the pendulum mass, ¢; is the pendulum length, J; is the pendulum 
inertia, and C| is a constant associated with friction (actual parameter values appear in [8]). 

For controller synthesis (and model linearization) we will require a state variable description of the 
system. This is easily done by defining state variables x; = 9, x2 60, x3 = 91, x4 = 61, and control 
signal u = vq. Linearization of these equations about the vertical position (i.e., 0; = 0), and using the 
system physical parameters [8] results in the following linear, time invariant state variable description: 


x] 0 1 0 0 x) 0 

Xp 0 —33.04 0 0 X 74.89 

_ (|= 55.23 
x3 0 0 0 1 x3 a 0 ( ) 
x4 0 49.30 73.41 —2.29 | [x4 —111.74 
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55.3.3 Swing-Up Control 


Because we intend to develop control laws that will be valid in regions about the vertical position (8; = 0), 
it is crucial to swing the pendulum up so that it is near vertical at near zero (angular) velocity. Elaborate 
schemes can be used for this task (such as those employing concepts from differential geometry), but 
for the purposes of this example we choose to use a simple heuristic procedure based on an “energy 
pumping strategy” proposed in [9] for a similar under-actuated system. The goal of this simple swing-up 
control strategy is to “pump” energy into the pendulum link in such a way that the energy or magnitude 
of each swing increases until the pendulum approaches its inverted position. To apply such an approach, 
we simply consider how one would (intuitively) swing the pendulum from its hanging position (6; = 1) 
to its upright position. If the rotating base is swung to the left and right continually at an appropriate 
frequency, the magnitude of the pendulum at each swing will increase. 

The control scheme we will ultimately employ consists of two main components: the “scheduler” 
(which we will also call a “supervisor”) observes the position of the pendulum relative to its stable 
equilibrium point (8; = m), then schedules the transitions between two reference positions of the rotating 
base (Q9"/ = +I); and, the “positioning control” regulates the base to the desired reference point. These 
two components compose a closed-loop planning algorithm to command the rotating base to move in 
a certain direction based on the position of the pendulum. In effect, the human operator acts as the 
supervisor in tuning the positioning control (through trial and error on the system). 

For simplicity, a proportional controller will be used as the positioning control. The gain Kp is chosen 
just large enough so that the actuator drives the base fast enough without saturating the control output; 
after several trials, Kp was set to 0.5. The parameter I’ determines how far the base is allowed to swing; 
larger swings transfer more energy to swinging up the pendulum. The swing-up motion of the pendulum 
can be approximated as an exponentially growing cosine function. The parameter I’ significantly affects 
the “negative damping” (i.e., exponential growth) of the swing-up motion. By tuning I’, one can adjust 
the motion of the pendulum in such a way that the velocity of the pendulum and the control output are 
minimum when the pendulum reaches its inverted position (i.e., the pendulum has the largest potential 
energy and the lowest kinetic energy). Notice that if the dynamics of the pendulum are changed (e.g., 
adding extra weight to the endpoint of the pendulum), then the parameter I must be tuned. In [8] it is 
shown how a rule-based system can be used to effectively automate the swing-up control by implementing 
fuzzy strategies in the supervisor portion of the overall scheme. 


55.3.4 Balancing Control 


Synthesis of the fuzzy controllers to follow is aided by (1) a good understanding of the pendulum dynamics 
(the analytical model and intuition related to the physical process), and (2) experience with performance 
of linear control strategies. Although numerous linear control design techniques have been applied to 
this particular system, here we consider the performance of only one linear strategy (the one tested) as 
applied to the experimental system: the linear quadratic regulator (LQR). Our purpose is twofold. First, 
we form a baseline for comparison to fuzzy control designs to follow, and second, we provide a starting 
point for synthesis of the fuzzy controller. It is important to note that extensive simulation results (on the 
nonlinear model) were carried out prior to application to the laboratory apparatus; designs were carried 
out on the linearized model of the system. Specifics of the design process for the LQR and other applicable 
linear design techniques may be found in other chapters of this volume. 

Because the linearized system is completely controllable and observable, state feedback strategies, 
including the optimal strategies of the LQR, are applicable. Generally speaking, the system performance 
is prescribed via the optimal performance index 


J= / * oxtt)? Ox(t) + ult) Ru(t)) dt, (55.24) 
0 
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where Q and R are the weighting matrices corresponding to the state x and input u, respectively. Given 
fixed Q and R, the feedback gains that optimize the function J can be uniquely determined by solv- 
ing an algebraic Riccati equation. Because we are more concerned with balancing the pendulum than 
regulating the base, we put the highest priority in controlling 6; by choosing the weighting matrices 
Q= diag(1,0,5,0) and R=[1]. For a 10 ms sampling time, the discrete optimal feedback gains cor- 
responding to the weighting matrices Q and R are k, = —0.9, ky = —1.1, k3 = —9.2, and kg = —0.9. 
Although observers may be designed to estimate the states 6; and 69, we choose to use an equally effective 
and simple first-order approximation for each derivative. 

Note that this controller is designed in simulation for the system as modeled (and subsequently 
linearized). When the resulting controller gains (k; through k4) are implemented on the actual system, 
some “trial-and-error” tuning is required (due primarily to modeling uncertainties), which amounted to 
adjusting the designed gain by about 10% to obtain performance matching the predicted results from 
simulation. Moreover, it is critical to note that the design process (as well as the empirical tuning) 
has been done for the “nominal” system (i.e., the pendulum system with no additional mass on the 
endpoint). 

Using a swing-up control strategy tuned for the nominal system, the results of the LQR control design 
are given in Figure 55.2 for the base angle (top plot), pendulum angle (center plot), and control output 
(bottom plot). 
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FIGURE 55.2. LQR on the nominal system. 
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55.4 Fuzzy Control for the Rotational Inverted Pendulum 


Aside from serving to illustrate procedures for synthesizing a fuzzy controller, several reasons arise for 
considering the use of a nonlinear control scheme for the pendulum system. Because all linear controllers 
are designed based on a linearized model of the system, they are inherently valid only for a region about 
a specific point (in this case, the vertical, 6; = 0 position). For this reason, such linear controllers tend to 
be very sensitive to parametric variations, uncertainties, and disturbances. This is indeed the case for the 
experimental system under study; when an extra weight or sloshing liquid (using a water-tight bottle) is 
attached at the endpoint of the pendulum, the performance of all linear controllers degrades considerably, 
often resulting in unstable behavior. Thus, to enhance the performance of the balancing control, one 
naturally turns to some nonlinear control scheme that is expected to exhibit improved performance 
in the presence of disturbances and uncertainties in modeling. Two such nonlinear controllers will be 
investigated here: in the next section, a direct fuzzy controller is constructed and later an adaptive version 
of this same controller is discussed. 


55.4.1 Controller Synthesis 


For simplicity, the controller synthesis example explained next will utilize singleton fuzzification and 
symmetric, “triangular” membership functions on the controller inputs and output (they are, in fact, very 
simple to implement in real-time code). We choose to use seven membership functions for each input, 
uniformly distributed across their universes of discourse (over crisp values of each input e;) as shown in 
Figure 55.3. The linguistic values for the ith input are denoted by EY where r € {—3, —2, —1,0, 1, 2, 3}. 
Linguistically, we would therefore define ES as “negative large,” Bie as “negative medium,” FE? as 
“zero,” and so on. Note also that a “saturation nonlinearity” is built in for each input in the membership 
functions corresponding to the outermost regions of the universes of discourse. We use min to represent 
the antecendent (i.e., “x” from Section 55.2 is min) and COG defuzzification. 

To synthesize a fuzzy controller for our example system, we pursue the idea of seeking to “expand” 
the region of operation of the fixed (nonadaptive) controller. In doing so, we will utilize the results of the 
LQR design presented in Section 55.3 to lead us in the design. A block diagram of the fuzzy controller 
is shown in Figure 55.4. Similar to the LQR, the fuzzy controller for the inverted pendulum system will 
have four inputs and one output. The four (crisp) inputs to the fuzzy controller are the position error of 
the base e1, its derivative e2, the position error of the pendulum e3, and its derivative e4. 

The normalizing gains g; essentially serve to expand and compress the universes of discourse to some 
predetermined, uniform region, primarily to standardize the choice of the various parameters in synthe- 
sizing the fuzzy controller. A crude approach to choosing these gains is strictly based on intuition and does 
not require a mathematical model of the plant. In that case the input normalizing gains are chosen in such 
a way that all the desired operating regions are mapped into [—1, +1]. Such a simple approach in design 
works often for a number of systems, as witnessed by the large number of applications documented in the 
open literature. For complicated systems, however, such a procedure can be very difficult to implement 
because there are many ways to define the linguistic values and linguistic rules; indeed, it can be extremely 
difficult to find a viable set of linguistic values and rules just to maintain stability. Such was the case for 
this system. 

What we propose here is an approach based on experience in designing the LQR controller for the 
linearized model of the plant, leading to a mechanized procedure for determining the normalizing gains, 
output membership functions, and rule-base. Recall from our discussion in Section 55.2 that a fuzzy 
system is a static nonlinear map between its inputs and output. Certainly, therefore, a linear map such as 
the LQR can be easily approximated by a fuzzy system (for small values of the inputs to the fuzzy system). 
Two components of the LQR are the optimal gains and the summer; the optimal gains can be replaced with 
the normalizing gains of a fuzzy system, and the summer can essentially be incorporated into the rule- 
base of a fuzzy system. By doing this, we can effectively utilize a fuzzy system implementation to expand 
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the region of operation of the controller beyond the “linear region” afforded by the linearization/design 
process. Intuitively, this is done by making the “gain” of the fuzzy controller match that of the LQR 
when the fuzzy controller inputs are small, while shaping the nonlinear mapping representing the fuzzy 
controller for larger inputs (in regions further from zero). 

As pointed out in Section 55.2, the rule-base contains information about the relationships between the 
inputs and output ofa fuzzy system. Within the controller structure chosen, recall that we wish to construct 
the rule-base to perform a weighted sum of the inputs. The summation operation is straightforward; prior 
to normalization, it is simply a matter or arranging each If... Then rule such that the antecedent indices 
sum to the consequent indices. Specification of the normalizing gains is explained next. 

The basic idea [10] in specifying the go — g4 is so that for “small” controller inputs (e;) the local slope 
(about zero) of the input-output mapping representing the controller will be the same as the LQR gains 
(ie., the k;). As alluded to above, the normalizing gains g; — g4 transform the (symmetric) universes of 
discourse for each input (see Figure 55.3) to [—1, 1]. For example, if [—f;, Bj] is the interval of interest 
for input i, the choice g; = 1/B; would achieve this normalization, whereas the choice gp = By would 
map the output of the normalized fuzzy system to the real output to achieve a corresponding interval of 
[—Bo, Bo]. Then, assuming the fuzzy system provides the summation operation, the “net gain” for the ith 
input-ouput pair is gjgo. Finally, therefore, this implies that gig9 = kj is required to match local slopes of 
the LQR controller and the fuzzy controller (in the sense of input-output mappings). 

We are now in position to summarize the gain selection procedure. Recalling (Section 55.3) that the 
optimal feedback gains based on the LQR approach are kj = —0.9, kp = —1.1, k3 = —9.2, and kg = —0.9, 
transformation of the optimal LQR gains into the normalizing gains of the fuzzy system is achieved 
according to the following simple scheme: 


¢ Choose the controller input which most greatly influences plant behavior and overall control objec- 
tives; in our case, we choose the pendulum position 0). Subsequently, we specify the operating range 
of the this input (e.g., the interval [—0.5, +0.5] radians, for which the corresponding normalizing 
input gain g3 = 2). 

¢ Given g3, the output gain of the fuzzy controller is calculated according to gp = k3/g3 = —4.6. 

¢ Given the output gain go, the remaining input gains can be calculated according to g; = k;/go, where 
j € {1,2,3, 4}, j Ai (note that i= 3). For go = —4.6, the input gains gj, g2, g3, and g4 are 0.1957, 
0.2391, 2, and 0.1957, respectively. 


Determination of the controller output universe of discourse and corresponding normalizing gain is 
dependent on the structure of the rule-base. A nonlinear mapping can be used to rearrange the output 
membership functions (in terms of their centers) for several purposes, such as to add higher gain near 
the center, to create a dead zone near the center, to eliminate discontinuities at the saturation points, 
and so on. This represents yet another area where intuition (i.e., knowledge about how to best control 
the process) may be incorporated into the design process. In order to preserve behavior in the “linear” 
region (i.e., the region near the origin) of the LQR-extended controller, but at the same time provide 
a smooth transition from the linear region to its extensions (e.g., regions of saturation), we choose an 
arctangent-type mapping to achieve this rearrangement. Because of the “flatness” of such a mapping near 
the origin, we expect the fuzzy controller to behave like the LQR when the states are near the process 
equilibrium. 

It is important to note that, unlike the input membership function of Figure 55.3, the output member- 
ship functions at the outermost regions of the universe of discourse do not include the saturating effect; 
rather, they return to zero value which is required so that the fuzzy controller mapping is well-defined. 
In general, for a fuzzy controller with n inputs and one output, the center of the controller output fuzzy 
set Y* would be located at where s = j + k+---+]is the index of the output fuzzy set Y* (and the output 
linguistic value), {j, k,..., 1} are the indices of the input fuzzy sets (and linguistic values), N is the number 
of membership functions on each input, and n is the number of inputs. Note that we must nullify the 
effect of divisor n by multiplying the output gain gop by the same factor. 
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55.4.2 Performance Evaluation 


55.4.2.1 Simulation 


Some performance evaluation via simulation is prudent to investigate the effectiveness of the strategies 
employed in the controller synthesis. Using the complete nonlinear model, the simulated responses of the 
direct fuzzy controller with seven membership functions on each input indicate that the fuzzy controller 
successfully balances the pendulum, but with a slightly degraded performance as compared to that of the 
LQR (e.g., the fuzzy controller produces undesirable high-frequency “chattering” effects over a bandwidth 
that may not be realistic in implementation). 

One way to increase the “resolution” of the fuzzy controller is to increase the number of membership 
functions. As we increase the number of membership functions on each input to 25, responses using 
the fuzzy controller become smoother and closer to that of the LQR. Additionally, the control surface 
of the fuzzy controller also becomes smoother and has a much smaller gain near the center. As a result, 
the control output of the fuzzy controller is significantly smoother. On the other hand, the direct fuzzy 
controller, with 25 membership functions on each input comes with increased complexity in design and 
implementation (e.g., a four-input, one-output fuzzy system with 25 membership functions on each input 
has 254 = 390,625 linguistic rules). 


55.4.2.2 Application to Nominal System 


Given the experience of the simulation studies, the final step is to implement the fuzzy controller (with 
seven membership functions on each input) on the actual apparatus. For comparative purposes, we 
again consider application to the nominal system, that is, the pendulum alone with no added weight or 
disturbances. With the pendulum initialized at its hanging position (0; = 1), the swing-up control was 
tuned to give the best swing-up response, as in the case of the LQR results of Section 55.3. The sampling 
time was set to 10 ms (smaller sampling times produced no significant difference in responses for any of 
the controllers tested on this apparatus). The only tuning required for the fuzzy control scheme (from 
simulation to implementation in experimentation) was in adjusting the value for g3 upward to improve 
the performance; recall that the gain g3 is critical in that it essentially determines the other normalizing 
gains. 

Figure 55.5 shows the results for the fuzzy controller on the laboratory apparatus; the top plot shows 
the base position (angle), the center plot shows the pendulum position (angle), and the bottom plot shows 
the controller output (motor voltage input). The response is comparable to that of the LQR controller 
(compare to Figure 55.2), in terms of the pendulum angle (ability to balance in the vertical position). 
However, some oscillation is noticed (particularly in the controller output, as predicted in simulation 
studies), but any difference in the ability to balance the pendulum is only slightly discernible in viewing 
the operation of the system. 


55.4.2.3 Application to Perturbed System 


When the system experiences disturbances and changes in dynamics (by attaching additional weight to the 
pendulum endpoint, or by attaching a bottle half filled with liquid), degraded responses are observed for 
these controllers. Such experiments are also informative for considerations of robustness analysis, although 
here we regard such perturbations on the nominal system as probing the limits of linear controllers (i.e., 
operating outside the linear region). 

As a final evaluation of the performance of the fuzzy controller as developed above, we show 
results when a container half-filled with water was attached to the pendulum endpoint. This essen- 
tially gives a “sloshing liquid” effect, because the additional dynamics associated with the sloshing liq- 
uid are easily excited. In addition, the added weight shifted the pendulum’s center of mass away from 
the pivot point; as a result, the natural frequency of the pendulum decreased. Furthermore, the effect 
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FIGURE 55.5 Direct fuzzy control on the nominal system. 


of friction becomes less dominant because the inertia of the pendulum increases. These effects obvi- 
ously come to bear on the balancing controller performance, but also significantly affect the swing-up 
controller as well. From the present chapter, we note that the swing-up control scheme requires tun- 
ing, once additional weight is added to the endpoint, and we refer the interested reader to [8] for 
details of a supervisory fuzzy controller scheme where tuning of the swing-up controller is carried out 
autonomously. 

With the sloshing liquid added to the pendulum endpoint, the LQR controller (and, in fact, other 
linear control schemes we implemented on this system) produced an unstable response (was unable to 
balance the pendulum). Of course, the linear control schemes can be tuned to improve the performance 
for the perturbed system, at the expense of degraded performance for the nominal system. Moreover, it 
is important to note that tuning of the LQR type controller is difficult and ad hoc without additional 
modeling to account for the added dynamics. Such an attempt on this system produced a controller with 
stable but poor performance. 

The fuzzy controller, on the other hand, because of its expanded region of operation (in the sense 
that it acts like the LQR for small inputs and induces a nonlinearity for larger signals), was able to 
maintain stability in the presence of the additional dynamics and disturbances caused by the sloshing 
liquid, without tuning. These results are shown in Figure 55.6 where some degradation of controller 
performance is apparent. Such experiments may also motivate the need for a controller, which can adapt 
to changing dynamics during operation; this issue is discussed later when we address adaptation in fuzzy 
controllers. 
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FIGURE 55.6 Direct fuzzy control on the pendulum with sloshing liquid at its endpoint. 


55.5 Adaptive Fuzzy Control 
55.5.1 Overview 


While fuzzy control has, for some applications, emerged as a practical alternative to classical control 
schemes, there exist rather obvious drawbacks. We will not address all of these drawbacks here (such as 
stability which is a current research direction); rather, we will focus on an important emerging topical area 
within the realm of fuzzy control, that of adaptive fuzzy control, as it relates to some of these drawbacks. 

The point that is probably most often raised in discussion of controller synthesis using fuzzy logic is that 
such procedures are usually performed in an ad hoc manner, where mechanized synthesis procedures, 
for the most part, are nonexistent (e.g., it is often not clear exactly how to justify the choices for many 
controller parameters, such as membership functions, defuzzification strategy, and inference strategy). On 
the other hand, some mechanized synthesis procedures do exist for particular applications, such as the one 
discussed above for the balancing control part of the inverted pendulum problem where a conventional 
LQR scheme was utilized in the fuzzy control design. Typically such procedures arise primarily out of 
necessity because of system complexity (such as when many inputs and multiple objectives must be 
achieved). Controller adaptation, in which a form of automatic controller synthesis is achieved, is one way 
of attacking this problem, when no other “direct” synthesis procedure is known. 

Another, perhaps equally obvious requirement in the design and operation of any controller (fuzzy or 
otherwise) involves questions of system robustness. For instance, we illustrated in our theme example that 
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the performance of the direct fuzzy controller constructed for the nominal plant may degrade if significant 
and unpredictable plant parameter variations, structural changes, or environmental disturbances occur. 
Clearly, controller adaptation is one way of overcoming these difficulties to achieve reliable controller 
performance in the presence of unmodeled parameter variations and disturbances. 

Some would argue that the solution to such problems is always to incorporate more expertise into 
the rule-base to enhance the performance; however, there are several limitations to such a philosophy, 
including: (1) the difficulties in developing (and characterizing in a rule-base) an accurate intuition about 
how to best compensate for the unpredictable and significant process variations that can occur for all 
possible process operating conditions; and (2) the complexities of constructing a fuzzy controller that 
potentially has a large number of membership functions and rules. Experience has shown that it is often 
possible to tune fuzzy controllers to perform very well if the disturbances are known. Hence, the problem 
does not result from a lack of basic expertise in the rule-base, but from the fact that there is no facility 
for automatically redesigning (i.e., retuning) the fuzzy controller so that it can appropriately react to 
unforeseen situations as they occur. 

There have been many techniques introduced for adaptive fuzzy control. For instance, one adaptive 
fuzzy control strategy that borrows certain ideas from conventional “model reference adaptive control” 
(MRAC) is called “fuzzy model reference learning control” (FMRLC) [11]. The FMRLC can automatically 
synthesize a fuzzy controller for the plant and later tune it if there are significant disturbances or process 
variations. The FMRLC has been successfully applied to an inverted pendulum, a ship-steering problem 
[11], antiskid brakes [12], reconfigurable control for aircraft [10], and in implementation for a flexible- 
link robot [13]. Modifications to the basic FMRLC approach have been studied in [14]. The work on the 
FMRLC and subsequent modifications to it tend to follow the main focus in fuzzy control where one 
seeks to employ heuristics in control. There are other techniques that take an approach that is more like 
conventional adaptive control in the sense that a mathematical model of the plant and a Lyapunov-type 
approach is used to construct the adaptation mechanism. Such work is described in [3]. There are many 
other “direct” and “indirect” adaptive fuzzy control approaches that have been used in a wide variety of 
applications (e.g., for scheduling manufacturing systems [15]). The reader should consult the references 
in the papers cited above for more details. 

Another type of system adaptation, where a significant amount and variety of knowledge can be 
loaded into the rule-base of a fuzzy system to achieve high-performance operation, is the supervisory 
fuzzy controller, a two-level hierarchical controller which uses a higher-level fuzzy system to supervise 
(coordinate or tune) a lower-level conventional or fuzzy controller. For instance, an expert may know 
how to control the system very well in one set of operating conditions, but if the system switches to 
another set of operating conditions, the controller may be required to behave differently to again achieve 
high-performance operation. A good example is the PID controller which is often designed (tuned) for 
one set of plant operating conditions, but if the operating conditions change the controller will not be 
properly tuned. This is such an important problem that there is a significant amount of expertise on how 
to manually and automatically tune PID controllers. Such expertise may be utilized in the development 
of a supervisory fuzzy controller, which can observe the performance of a low-level control system and 
automatically tune the parameters of the PID controller. Many other examples exist of applications where 
the control engineer may have a significant amount of knowledge about how to tune a controller. One 
such example is in aircraft control when controller gains are scheduled based on the operating conditions. 
Fuzzy supervisory controllers have been used as schedulers in such applications. In other applications we 
may know that conventional or fuzzy controllers need to be switched on based on the operating conditions 
(see the work in [16] for work on fuzzy supervision of conventional controllers for a flexible-link robot) 
or a supervisory fuzzy controller may be used to tune an adpative controller (see the work in [10] where 
a fuzzy supervisor is used to tune an adaptive fuzzy controller that is used as a reconfigurable controller 
for an aircraft). It is this concept of monitoring and supervising lower-level controllers (possibly fuzzy, 
possibly conventional) that defines the supervisory control scheme. Indeed, in this sense supervisory and 
adaptive systems can be described as special cases of one another. 
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55.5.2 Auto-Tuning for Pendulum Balancing Control 


Many techniques exist for automatically tuning a fuzzy controller in order to meet the objectives men- 
tioned above. One simple technique we present next, studied in [8,14,15], expands on the idea of increasing 
the “resolution” of the fuzzy controller in terms of the characteristics of the input membership functions. 
Recall from Section 55.4 for our theme problem that when we increased the number of membership func- 
tions on each input to 25, improved performance (and smoother control action) resulted. Likewise, we 
suspect that increasing the resolution would result in improved performance for the perturbed pendulum 
case. 

To increase the resolution of the direct fuzzy controller with a limited number of membership func- 
tions (as before, we will impose a limit of seven), we propose an “auto-tuned fuzzy control.” To gain 
insight on how the auto-tuned fuzzy control works, consider the idea of a “fine controller,” with smooth 
interpolation, achieved using a fuzzy system where the input and output universes of discourse are nar- 
row (i.e., the input normalizing gains are large, and the output gain is small). In this case there are many 
membership functions on a small portion of the universe of discourse (i-e., “high resolution”). Intuitively, 
we reason that as the input gains are increased and the output gain is decreased, the fuzzy controller 
will have better resolution. However, we also conjecture that to obtain the most effective control action 
the input universes of discourse must also be large enough to avoid saturation. This obviously raises 
a question of trying to satisfy two opposing objectives. The answer is to adjust the gains based on the 
current operating states of the system. For example, if the states move closer to the center, then the input 
universe of discourse should be compressed to obtain better resolution, yet still cover all the active states. 
If the states move away from the center, then the input universe of discourse must expand to cover these 
states, at the expense of lowering the resolution. 

The input-output gains of the fuzzy controller can be tuned periodically (e.g., every 50 samples) using 
the auto-tuning mechanism shown in Figure 55.7. Ideally, the auto-tuning algorithm should not alter the 
nominal control algorithm near the center; we therefore do not adjust each input gain independently. 
We can, however, tune the most significant input gain, and then adjust the rest of the gains based on this 
gain. For the inverted pendulum system, the most significant controller input is the position error of the 
pendulum, e3 = 0). 

The input-output gains are updated every n; samples in the following manner: 


¢ Find the maximum e3 over the most recent n, samples and denote it by e}'™*. 

¢ Set the input gain g3 = 1/|e}™™*|. 

¢ Recalculate the remaining gains using the technique discussed in Section 55.4 so as to preserve the 
nominal control action near the center. 


We note that the larger n, is, the slower the updating rate is, and that too fast an updating rate may cause 
instability. Of course, if a large enough buffer were available to store the most recent n,; samples of the 
input, the gains could be updated at every sample (utilizing an average); here we minimized the usage of 
memory and opted for the procedure mentioned above (finding the maximum value of e3). 

Simulation tests (with a 50-sample observation window and normalizing gain g3 = 2) reveal that when 
the fuzzy controller is activated (after swing up), the input gains gradually increase while the output gain 
decreases, as the pendulum moves closer to its inverted position. Asa result, the input and output universes 
of discourses contract, and the resolution of the fuzzy system increases. As g3 reaches its maximum value”, 
the control action near 9; = 0 is smoother than that of direct fuzzy control with 25 membership functions 
(as investigated previously via simulation), and very good balancing performance is achieved. 

When turning to actual implementation on the laboratory apparatus, some adjustments were done 
in order to optimize the performance of the auto-tuning controller. As with the direct fuzzy con- 
troller, the value of g3 was adjusted upward, and the tuning (window) length was increased to 75 


* In practice, it is important to constrain the maximum value for g3 (for our system, to a value of 10) because disturbances 


and inaccuracies in measurements could have have adverse effects. 
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FIGURE 55.7 Auto-tuned fuzzy control. 


vC-SS 


spoylay paouvapy urta1s4§ [omu0D 


Fuzzy Control 55-25 


sg 2 T T T T T T T T T 
g 

vo 

E 

Sa 0 

g 

— 

a) ! L D ! L ! ! ! 1 

0 1 2. 3 4 5 6 7 8 9 10 


Time (s) 


Position of pendulum (rad) 


5 L L L L L L | | 1 
0 1 2 3 4 5 6 7 8 9 10 


Time (s) 


Control output (volts) 


FIGURE 55.8 Auto-tuned fuzzy control on the pendulum with sloshing liquid at its endpoint. 


samples. The first test was to apply the scheme to the nominal system; the auto-tuning mechanism 
improved the response of the direct fuzzy controller (Figure 55.5) by varying the controller resolution 
online. That is, as the resolution of the fuzzy controller increased over time, the high-frequency effects 
diminished. 

The true test of the adaptive (auto-tuning) mechanism is to evaluate its ability to adapt its con- 
troller parameters as the process dynamics change. Once again we investigate the performance when 
the “sloshing liquid” dynamics (and additional weight) are appended to the endpoint of the pendulum. 
As expected from simulation exercises, the tuning mechanism, which “stretches” and “compresses” the 
universes of discourse on the input and output, not only varied the resolution of the controller but also 
effectively contained and suppressed the disturbances caused by the sloshing liquid, as clearly shown in 
Figure 55.8. 


55.6 Concluding Remarks 


We have introduced the general fuzzy controller, shown how to design a fuzzy controller for our rotational 
inverted pendulum theme problem, and compared its performance to a nominal LQR controller. We 
overviewed supervisory and adaptive fuzzy control techniques and presented a particular adaptive scheme 
that worked very effectively for the balancing control of our rotational inverted pendulum theme problem. 
Throughout the chapter we have emphasized the importance of comparative analysis of fuzzy controllers 


55-26 Control System Advanced Methods 


with conventional controllers with the hope that more attention will be given to detailed engineering 
cost-benefit analyses rather than the hype that has surrounded fuzzy control in the past. 

There are close relationships between fuzzy control and several other intelligent control methods. 
For instance, expert systems are generalized fuzzy systems since they have a knowledge-base (a gener- 
alized version of a rule-base where information in the form of general rules or other representations 
may be used), an inference mechanism that utilizes more general inference strategies, and are con- 
structed using the same general approach as fuzzy systems. There are close relationships between some 
types of neural networks (particularly radial basis function neural networks) and fuzzy systems and, 
while we did not have the space to cover it here, fuzzy systems can be trained with numerical data 
in the same way that neural networks can (see [3] for more details). Genetic algorithms provide for a 
stochastic optimization technique and can be useful for computer-aided-design of fuzzy controllers, 
training fuzzy systems for system identification, or tuning fuzzy controllers in an adaptive control 
setting. 

While there exist such relationships between fuzzy control and other techniques in intelligent control, 
the exact relationships between all the techniques have not been established. Research along these lines 
is progressing but will take many years to complete. Another research area that is gaining increasing 
attention is the nonlinear analysis of fuzzy control systems and the use of comparative analysis of fuzzy 
control systems and conventional control systems to determine the advantages and disadvantages of fuzzy 
control. Such work, coupled with the application of fuzzy control to increasingly challenging problems, 
will help establish the technique as a viable control engineering approach. 


55.7 Defining Terms 


Rule-base: A part of the fuzzy system that contains a set of If... Then rules that quantify a human expert’s 
knowledge about how to best control a plant. It is a special form of a knowledge-base that only 
contains If... Then rules that are quantified with fuzzy logic. 

Inference Mechanism: A part of the fuzzy system that reasons over the information in the rule-base 
and decides what actions to take. For the fuzzy system the inference mechanism is implemented 
with fuzzy logic. 

Fuzzification: Converts standard numerical fuzzy system inputs into a fuzzy set that the inference 
mechanism can operate on. 

Defuzzification: Converts the conclusions reached by the inference mechanism (i.e., fuzzy sets) into 
numeric inputs suitable for the plant. 

Supervisory Fuzzy Controller: A two-level hierarchical controller which uses a higher-level fuzzy system 
to supervise (coordinate or tune) a lower level conventional or fuzzy controller. 

Adaptive Fuzzy Controller: An adaptive controller that uses fuzzy systems either in the adaptation 
mechanism or as the controller that is tuned. 
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Further Reading 


Other introductions to fuzzy control are given in [2-5]. The book [2] is available for a free download 
(as a .pdf file) from http://www.ece.osu.edu/“passino/. To study other applications of fuzzy control the 
reader should consult the references in the papers and books cited above. There are several conferences 
that have sessions or papers that study various aspects of fuzzy control. For instance, see the Proceedings 
of FuzzIEEE (in the World Congress on Computational Intelligence), IEEE International Symposium on 
Intelligent Control, Proceedings of the IEEE Conference on Decision and Control, the American Control 
Conference, and many others. The best journals to consult are the IEEE Transactions on Fuzzy Systems, 
Fuzzy Sets and Systems, and the IEEE Transactions on Systems, Man, and Cybernetics. Also, there are 
many short courses that are given on the topic of fuzzy control. 
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56.1 Introduction 


In control, filtering, and prediction applications the available a priori model information is sometimes 


not sufficiently accurate for a fixed design to satisfy all of the design specifications. In these circumstances, 


the designer may 


¢ Reduce the specified level of required performance. 
¢ Expend additional efforts to reduce the level of model uncertainty. 
¢ Design a system to adjust itself online to increase its performance level as the accumulated input 


and output data are used to reduce the amount of model uncertainty during operation. 


Controllers (filters and predictors can be implemented by similar techniques) that implement the last 


approach are the topic of this chapter. Such techniques are suitable when additional a priori effort in 


modeling and validation is unacceptable because of cost and feasibility, and the desired level of perfor- 


mance must be maintained. 


Adaptive linear control mechanisms (see [17] and Chapters 34 and 35 of this book) have been developed 
for systems that, by suitable limitation of their operating domain, can be adequately modeled as linear. 
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The online system identification and control techniques developed for nonlinear systems can be classified 
into two categories, parametric and nonparametric. Parametric techniques assume that the functional 
form of the unknown nonlinearity is known, based on physical modeling principles, but that the model 
parameters are unknown. Nonparametric techniques are necessary when the functional form of the 
unknown nonlinearity is itself unknown. In this case a general family of function approximators is selected 
based on the known properties of the approximation family and the characteristics of the application. 
Similar to parametric techniques, the objective of the nonparametric approach is still to select a set of 
parameters to achieve an optimal approximation of the unknown function. 

Learning and neural control (and system identification) [13,14,36] are nonparametric techniques 
developed to enhance the performance of those poorly modeled nonlinear systems for which a suitable 
model structure is not available. This performance improvement is achieved by exploiting experiences 
obtained through online interactions with the actual plant. Adaptive approximation-based control is an 
alternative name used in the literature for this class of techniques, and it includes approximators such as 
neural networks, fuzzy systems and traditional approximation approaches. Neural control is the name 
applied in the literature when a neural network is selected as the function approximator. To be both concise 
and general, adaptive approximation-based control will be the name used throughout this chapter. 

Adaptive approximation-based control has an important role to play in the development of advanced 
control systems. This class of techniques has become feasible in recent decades due to the rapid advances 
that have occurred in computing technologies. Inexpensive desktop computing has inspired many ad hoc 
approximation-based control approaches. In addition, similar approaches in different communities (e.g., 
neural, fuzzy) have been derived and presented using different nomenclature yet nearly identical theoreti- 
cal results. Our objective in this chapter is to treat such approaches within a unifying framework so that 
the resulting presentation encompasses a variety of approximation structures with different properties. 

The three terms, adaptation, learning, and self-organization, are used with different meanings by 
different authors in the literature. In this chapter, we will use adaptation to refer to temporal changes. 
For example, adaptive control is applicable when the estimated parameters are slowly varying functions 
of time. We will use learning to refer to methods that retain information as a function of measured 
variables. Herein, learning is implemented via function approximation. Therefore, learning has a spatial 
connotation whereas adaptation refers to temporal effects. The process of learning requires adaptation, 
but the retention of information as a function of other variables in learning implies that learning is 
a higher level process than is adaptation. Self-organization refers to methods that adapt the function 
approximation structure during online operation [11,38]. Such methods are not covered in this chapter. 

This chapter will focus on the motivation for, and implementation of, adaptive approximation-based 
control systems. Many of the concepts and results from function approximation and data interpolation 
are important relative to adaptive approximation-based control. Section 56.2 discusses various properties 
of function approximators as they relate to adaptive function approximation for control purposes. Section 
56.3 presents a few function approximation structures that have been considered for implementation of 
adaptive approximation-based controllers. Section 56.4 focuses on issues related to parameter estimation. 
Section 56.5 reviews various nonlinear control system design methodologies. In Section 56.6 we bring 
together the ideas of function approximation, parameter estimation, and nonlinear control in the design of 
adaptive approximation-based control systems. Section 56.7 presents briefly the concept of approximate 
dynamic programming (ADP), which constitutes an important class of neural control schemes that has 
attracted a lot of attention in the literature. Finally, Section 56.8 contains some concluding remarks. 


56.2 Approximator Properties 


This section discusses issues and trade-offs related to the selection of a function approximation structure. 
We consider the situation where a control problem and solution have been defined that include a function 
h(z). If the function h were known, then the control approach could be directly implemented. When 
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h is unknown, then the control law is rewritten using h(z;), which is intended to approximate the 
unknown function h(z) well for some optimal value 6 = 0*, which will be estimated online. In this 
discussion, the symbol z represents a dummy vector variable that may contain portions of the state vector 
or exogenous input variables. To proceed, it is necessary that the vector z be known or measured. We 
are interested in those applications where h(z) is unknown, but we have information available to us that 
allows approximation of h(z) for z € D, where D is a domain of interest. 

A few examples are useful to clarify the meaning of the function h and the argument z. For example, 
h(z) could represent the system dynamics f(x, u) where z would then represent the vector [x, u]. In an 
adaptive gain scheduling approach, h(z) might represent a mapping from the current operating condition 
represented by z to linear control system parameters. 

It is almost always desirable to have theoretical results showing that the control law performance using 
h(z; 6) is asymptotically equivalent to that of the control law using h(z). From the control theory point of 
view, it is necessary to show boundedness of all signals in the control loop and asymptotic convergence 
of certain signals (e.g., tracking errors) toward zero. Preferably, the previously mentioned bounds can be 
made as small as desired, without increasing the control system gains (i.e., bandwidth). In addition, it is 
sometimes of interest to analyze whether h(z; Q) converges to h(z) as t > oo for all z € D. It should be 
noted that the control performance goal can be achieved without convergence of the approximator. 

Note that we have converted the function approximation problem to a parameter estimation problem, 
by assuming a known structure for the approximator. Selecting this structure is a critical design decision. 


56.2.1 The Domain of Approximation 


Whether convergence of the function approximator is required or not, proper definition of the function 
approximation problem is important. The optimal function approximator will minimize some measure 
of the error between hz; 9) and h(z). Although the function h(z) may be defined for z € 3”, the function 
approximation problem can only be well defined over some compact set denoted by D. For example, the 
problem of finding 


0* = argmino] (8), (56.1) 


where 


(0) = [ \n(z) — h(2s0) |? dz (56.2) 


is well defined* for compact D and h, h € C(D).To see the necessity of the constraint that the optimization 
be over a compact set D, consider the simple problem of finding the optimal constant approximation 
h(z : 8) = Oto the quadratic function h(z) = z*. While this problem has a well-defined solution over every 
compact subset of it, there is no solution over the set of real numbers ‘i. 

While a few papers fail to state assumptions on the domain of approximation D, the vast majority of 
the papers in the literature work with a fixed domain of approximation that is defined a priori. At this 
point in time, the authors are unaware of articles that define the region D during operation in a rigorous 
fashion, which is an issue closely related to the topic of self-organization of the function approximation 
structure. 

Finally, with the approximator defined over a domain D, the designer must be concerned with two 
situations related to the time evolution of the argument z(t) as a function of time. First, the initial value 
z(0) may not be in D. In this case, the designer must ensure that z(t) enters D in finite time so that the 
benefits of the adaptive approximation can be realized. Second, once z(t) enters D, the designer must 
ensure either that it does not leave D or that it returns to D in finite time [32]. 


* The notation C(D) is read as “the set of functions continuous on domain D.” 
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56.2.2 Approximation Error 
When h(z) € C(D) and h(z; 8) € C(D) for all 0, then for any 0, the approximation error 


€(z;0) = h(z) — h(z;9) (56.3) 


is continuous and bound for z € D. Therefore, the minimization defined in Equation 56.2 is well defined 
and the optimal error function 


e*(z) = €(z;0*) = h(z) — h(z;6*) (56.4) 


is also a continuous bounded function for z € D. The function €*(z) will be referred to as the minimum 
function approximation error (MFAE). 
Many articles in neural control include an assumption similar to the following. 


Assumption 56.1: 


For the functions h(z) € C(D) and A(z; 8) € C(D), there exists 0* € R? and € > 0 such that ||e(z; 0*)|| < € 
VzeED. 


The boundedness of the approximation error—for a continuous function h(z), a continuous approx- 
imator h(z;6), and a compact domain D—is a property of the problem statement. The assumption is 
only giving a name € to the bound. Ideally, the designer expects the bound € to be small, but the size 
of the bound will be directly affected by the designer’s choice of the function approximation structure 
h(z;). Because the approximation error may significantly affect the control performance, the choice of 
the approximation structure is very important. 


56.2.3 The Context 


Function approximation problems are well studied, especially in the case where h(z) and h(z;) are known 
analytically and D is given. Examples include the decomposition of a known function into its Fourier or 
Taylor series representation. Such problems are often studied in purely mathematical contexts [22,29]. 

There is also vast literature related to the development of functions to either interpolate or approximate 
a given batch of data. Online approximation applications are different in that the data are not a fixed 
batch, but are arriving continuously as the system operates. In addition, there is no guarantee that the 
data that arrive will represent the region D in a statistically uniform sense. Such problems are sometimes 
referred to as incremental, scattered data, approximation problems. 

In the applications that are of interest to this chapter, h(z) is not known as an analytic formula; 
therefore, Equation 56.2 cannot be optimized analytically. Instead, a typical approach is to optimize a cost 
function based on online measurements that depend on the function (i.e., h(z(t))), where z(t) is defined 
by the system trajectory. When the values of h(z(t)) are available as a function of time, the cost function 


is effectively 
1 t 
§) = - 
10) == f 


Note, that while the cost function of Equation 56.2 is uniformily weighted over the region D, the cost 
function of Equation 56.5 is biased toward those portions of D wherein the system operates most often. 
Therefore, the two cost functions can have significantly different minimizing values of 0. 

Finally, it is often the case that the values of h(z(t)) are not directly available; therefore, even Equa- 
tion 56.5 cannot be directly evaluated. Instead, adjustments to the parameter estimate 6 will be inferred, 
possibly based on stability-based formulations using the tracking error or identification error. 


h(z(1)) — h(z(2);8) |’ dz. (56.5) 
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56.2.4 Active and Passive Learning 


The fact that the sample locations z(t) € D are determined by the system trajectory leads to an interesting 
distinction between active and passive learning scenarios. Active learning describes those situations in 
which the designer has the freedom to control the training sample distribution. Passive learning describes 
those situations in which the training sample density is defined by some external mechanism. Online 
applications usually involve passive learning because the plant is performing some useful function. 

When active learning is possible, the designer will have the additional task to define the trajectory z(t) 
to appropriately trade-off control performance relative to achieving accurate function approximation 
over D. 


56.2.5 Local versus Global Models 


In the selection of a suitable model structure, it is useful to consider the domain over which the model is 
expected to apply. In particular, the following two definitions are of interest. 


Definition 56.1: Local Approximation Structure 


A parametric model h(z; 0) is a local approximation to h(z) at zo € D if, for any € > 0, there exist 8 and} 
so that ||h(z) — h(z;8)|| < ¢ for all z € B(zo, 8) where B(zo, 8) = {z | ||z — Zoll < 8}. 


Definition 56.2: Global Approximation Structure 


A parametric model h(z;0) is a global approximation to h(z) over domain D if for any ¢ > 0 there exists 0 
so that ||h(z) — h(z;8)|| < ¢ for allz € D. 


The following items are of interest relative to the definition of local and global models: 


¢ Physical models derived from first principles are expected to be global models (i.e., valid over the 
domain of operation D). 

¢ Whether a given approximation structure is local or global depends on the system that is being 
modeled and the domain D. 

« Ifa parameter vector 0 exists satisfying Definition 56.2 for a particular ¢, then this 9 also satisfies 
Definition 56.1 for the same ¢ at each xp € D. Therefore, the set of global models is a strict subset 
of the set of local models. 

¢ As the desired level of accuracy increases (i.e., ¢ decreases), the dimension of the parameter vector 
or the complexity of the model structure will usually have to increase. 


56.2.6 Adaptation versus Learning 


As the operating point z(t) moves through D, a local approximation structure could maintain accuracy 
in the vicinity of the operating point by adjusting the parameter vector through time (i.e., adaptation). 
Alternatively, the parameter vector could be stored as a function of the operating point (ie., learning). The 
former approach is typical of linear adaptive control methodologies in nonlinear control applications. 
The latter approach is one implementation of learning control [12] that constructs a global approximation 
structure by connecting several local approximating structures together in a smooth fashion [3]. 

The conceptual differences between adaptive linear control and learning control are of interest. In 
a nonlinear application, adaptive linear control methodologies can be developed to maintain a locally 
accurate model or control law at the current operating point zo. With the assumption that the linearized 
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model will change over time, as the operating point changes, the adaptive mechanism will adjust the 
parameter estimates to maintain the accuracy of the local fit. Such an approach can provide satisfactory 
performance if the locally linearized model changes slowly, but will have to estimate the same model 
parameters repetitively if the operating point moves repetitively over a set of standard operating points. 
A more ambitious proposal is to develop a global model of the dynamics that stores model information 
locally as a function of operating condition. One mechanism for this approach is to store the local linear 
models or controllers as a function of the operating point. 
For example, 


i(z;0) =) (Adz —x) 4 a) w(t), (56.6) 
i=1 


where jt > 0; w(K): Rt b> R* is continuous and positive function that is non-zero only for i € [0, 1] 
with )7i", w (=) = 1 forallz € D;and Aj € #”*” and B; € R” are unknown parameters. Given these 


definitions, with appropriately chosen parameters, the function Aj(z — zj;) + B; is the linearized model 
at operating point z;. The support of each linearized model is S; = B(z;, 4). By the assumptions stated, 
DC UZ, S;. If 4 is selected small enough so that each linear model achieves approximation accuracy € 
over its support region, then the approximator h defined in Equation 56.6 will achieve accuracy € over D. 
If a parameter adjustment mechanism is designed such that A; and B; are only adjusted for z € S;, then 
the model will be adjusted locally and retained locally; therefore, learning (i.e., retention of information 
within a recallable context) will have been achieved. 

The name Learning Control stems from the idea that past performance information is used to estimate 
and store information (relevant to an operating point) for use when the system operates in the vicinity 
of the same operating point in the future. Learning Control requires more extensive use of computer 
memory and computational effort than adaptive or fixed parameter control. The required amount of 
memory and computation will be determined predominantly by the selection of function approximation 
and parameter estimation (training) techniques. 


56.2.7 Universal Approximation 


Given the problem of approximating the unknown function h(z) € C(D) over the compact domain D, the 
structure of the approximator h(z;®) should be carefully considered. Ideally, the designer should know 
that if the dimension of 0 is increased, in some appropriate way, then it is possible to make 


max |e*(z)| = max |h(z) — A(z; 6*)| 
zEeD zEeD 


sufficiently small. Such results are referred to as universal approximation (UA) theorems and the approx- 
imators that satisfy the theorems are referred to as universal approximators. 

Several UA results have been derived in the literature, for example, [16,26]. A typical result is presented 
below for discussion. Prior to the statement of the theorem, we require a definition for ZIT networks. 


Definition 56.3: 


The family of r input, N node, single hidden layer (XII) network approximators associated with nodal 
processor g(-) is defined by 


Sw = ( h(z;0,w,b)= > 0; | |g (wyz a bi) 


F wy 


q 
i=1 j= 
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Theorem 56.1: [16] 


Let D CR" be compact and g : Rt > N be any continuous nonconstant function. Then, the set Syn with 
nodal processor g has the property that for any h € C(D) and any € > 0, there exists N (sufficiently large) 
such that maxzep |e*(z)| <. 


This theorem indicates that if g is a nodal function satisfying certain technical assumptions and ¢ is 
a specified approximation accuracy, then any given continuous function can be approximated over the 
compact set D to the desired degree of accuracy if the number of nodes N is sufficiently large. Such UA 
results are powerful, but should be interpreted with caution. 

First, such theorems are very general. They do not state that any specific type of neural network has 
superior abilities for function approximation. In fact, the constraints on the nodal function g are quite lax. 

Second, UA requires that the number of nodes per layer be expandable. In most applications, the 
number of nodes N is fixed a priori. Once each N is fixed, the network can no longer approximate an 
arbitrary continuous function to a specified accuracy €; hence, approximation structures that allow criteria 
for the network structure specification are beneficial. Also, UA is achieved by making N “sufficiently” 
large. Starting a priori with a small N defeats the objective of using a universal approximator. 

Third, UA results state existence. They are not constructive. In particular, neither the required value 
of N for a particular nodal processor, nor the approximator structure are defined by the theorems. 


56.2.8 Generalization 


For the applications discussed, the parameters of the approximation structure will be estimated using 
the training set 7; = {z(t) for t € [0, t]} defined by the system trajectory. However, the approximating 
function may be evaluated at any point z € D. Therefore, it is desirable that the resulting approximation 
converge to a form that is accurate throughout D, not only at the points z € 7;. The ability of parametric 
approximators to provide answers—good or bad—at points outside the training set is referred to in the 
neural network literature as generalization. 

The fact that an approximation structure is capable of generalizing from the training data is not neces- 
sarily a beneficial feature. Note that the approximator will output results h(z;0) at any evaluation point z. 
The user must take appropriate steps to verify or ensure the accuracy of the resulting approximation at 
that point. Families of approximators that allow this assessment to occur online are desirable. 

Generalization of training data may also be described as interpolation or extrapolation of the training 
data. Interpolation refers to the process of filling in the holes between nearby training samples. Interpo- 
lation is a desirable form of generalization. Most approximators interpolate well. Some approximators 
will allow online analysis of the interpolation accuracy. Extrapolation refers to the process of providing 
answers in regions of the learning domain D that the training set 7; does not adequately represent. Extrap- 
olation from the training data is risky when the functional form is unknown by assumption. Clearly, it is 
desirable to know whether the approximator is interpolating between the training data or extrapolating 
from the training data. 


56.2.9 Parameter (Non)Linearity 


A general representation of function approximators is 


h(z; 9,0) = o(z30) "8, (56.7) 
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where the adjustable parameters are represented by 9, which appears linearly, and o, which appears 
nonlinearly. For example, in the approximator 


N 
h(z; 0,0) = Ss 0; exp (= (z _ oi)’) F (56.8) 
i=l 


the coefficients 6; appear linearly while the center location parameters o; appear nonlinearly. The design 
must determine whether to adjust both sets of parameters or just those that appear linearly. If all the non- 
linear parameters are fixed during the design stage, then a linear-in-the-parameters (LIP) approximator 
is obtained. Nonlinear in the parameter approximators are more flexible, but also are more difficult to 
analyze theoretically and to tune in an online fashion. 

The relative drawbacks of LIP approximators are discussed, for example, in [4] where it is shown that 
under certain technical assumptions, nonlinear in the parameter approximators have squared errors of 
order © (1/N), whereas LIP approximators cannot have approximation errors less than O (1/N a ") (N 
is the number of parameters and n is the dimension of D). 

Function approximators that are LIP can be represented as 


h(z;0) = (z)'®, (56.9) 


where 9 is the unknown parameter vector, and the regressor vector ¢(z) is a known, usually nonlinear, 
vector function of z. The powerful parameter estimation and performance analysis techniques that exist 
for LIP approximators (see [15]) make this a beneficial property. 

By substituting Equation 56.3 into Equation 56.2, we obtain 


J(8) -| lle(z; 9)||? dz, (56.10) 
D 


which is strictly convex in €. For LIP approximators, defining 6 = @ — 6*, by Equations 56.3 and 56.4 it is 
clear that 
e(z; 0) = —o(z) 6+ €*(2), 

which is linear in the parameter error vector. Therefore, J(9) is convex in the parameter error vector. Note 
that for a fixed value of z, there is a linear subspace of values © defined as © = {0 : o(z)'0 = 0}. If z 
varies sufficiently, then this subspace will shrink down to the single point 6*. The fact that J is convex in «, 
which is linear in 0, ensures that © is the unique equivalence set of global minima. When the parameters 
appear in a nonlinear fashion, then numerous local minima may also exist. 

As discussed in Section 56.2.3, when a sample error cost function such as Equation 56.5 is used in 
place of the cost function of Equation 56.2, the optimal parameter estimate that results will depend on 
the distribution of the training samples in the training set 7;. The fact that different parameter estimates 
result from different sets of training samples is not the result of multiple local minima; it is, instead, the 
result of different weightings by the sample distribution in the cost function. 


56.2.10 Coverage 


Coverage ensures that the value of the approximator can be adjusted at any point z € D. This property 
can be stated formally as follows: for any z € D, there exists at least one 0; such that |Oh(z; 6;)/00| 4 0. If 
this property does not apply, then there exists some point z € D for which the approximation cannot be 
changed. This is obviously an undesirable situation. 


56.2.11 Localization 


The localization property is stated formally as follows: for 9;, if | Oh(z; Q)/ 30; is nonzero in the vicinity of 
Zo € D, then it must be zero outside the ball B(z,, 8) for some § > 0. The smallest such 8 is the radius of 
support. 
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With the localization property, the effects of changing any single parameter are limited to radius of 
support of that parameter. Thus, experience and consequent learning in one part of the input domain 
cannot positively nor negatively affect the knowledge previously accrued in other parts of the mapping 
domain. A polynomial or Fourier series approximation does not have the localization property. It is easy 
to see that adjusting any parameter of such an approximator will affect the approximation throughout D. 

The fact that a limited subset of the model parameters is relevant for a given point in the learning 
domain also makes it possible to reduce significantly the required amount of computation per sample 
interval. The trade-off is that an approximator with the localization property may require a higher number 
of parameters (more memory) than an approximator without the property. For example, if the domain D 
has dimension d, and m parameters are necessary for a given local approximator per input dimensions, 
then on the order of m@ parameters will generally be required to approximate the D dimensional function. 
This exponential increase in the memory requirements with input dimension is referred to in the literature 
as the curse of dimensionality.” 


56.3 Function Approximators 


Examples of several classes of function approximators are given in this section. This list of examples is 
not meant to be comprehensive. Many additional classes of approximators exist. See for example [13]. 


56.3.1 Sigmoidal Neural Networks 


A strict mathematical definition of what is or is not a neural network does not exist; however, the class of 
sigmoidal neural networks has become widely accepted as function approximators. 
Single hidden-layer neural networks have the form 


N 
h(z) = > 9; (ojo = [of, wae ,0jM]Z)s (56.11) 
j=l 


where usually both the linear parameters 6 ¢ 3% and the nonlinear parameters o ¢ RN*™+) are 
adjusted, and the nodal processor ¢ is a scalar function of a single variable. Most often, the nodal 
processor has a sigmoidal shape similar to o(v) = atan(v) or o(v) = 1/(1+e7”). 

Multiple layer networks are constructed by cascading single layer networks together, with the output 
from one network serving as the input to the next. 


56.3.2 Orthogonal Polynomial Series 


A set of polynomial functions $;(z),i=1,2,... is orthogonal with respect to a nonnegative weight 
function w(t) over the interval [a, b], if 


b es 
[ warviono d= | ie 


for some nonzero constants r;. When a function h(z) satisfies certain integrability conditions, it can be 
expanded as 


h(z) = D> ndn(z)s 
n=0 


* The term curse of dimensionality was originally used by Bellman to refer to the increasing complexity of the solution to 


dynamic programming problems with increasing input dimension. 


56-10 Control System Advanced Methods 


where 


b 
On = =| w(z)h(z)oj(z) dz. 


The mth order finite approximation of h(z) with respect to the polynomial series $;(z) is given by 
7 m—1 
h(z) = D> Ondn(z). (56.12) 
n=0 


The integral of the error between h(z) and h(z) over [a, b] converges to zero as m approaches oo. When 
h(z) is unknown, it may be reasonable to approximate the 0, in Equation 56.12 using online data. 


56.3.3 Localized Basis Influence Functions 


Due to the usefulness of the interconnection of local models to generate global models, the class of Basis 
Influence Functions [25] is presented. The definition is followed by examples of several approximation 
architectures that satisfy the definition. The purpose of the examples is to demonstrate the concept and 
to illustrate that several approximators often discussed independently can be studied as a class within the 
setting of the definition. 


Definition 56.4: Localized-Basis Influence Functions 


A function approximator is of the BI Class if, and only if, it can be written as 


h(zs8) = S~ hi(zs 4, gi) i(zszi), (56.13) 


where each h;(z; 6, zi) is a local approximation to h(z) for allz € B(z;,8); P'(z; 2) has local support S; = {z : 
T(z; zi) 4 0}, which is a subset of B(z;,8); and D C U; Si. 


In this framework, the h;(z; 6, z;) are the basis functions and the I’;(z; z;) are the influence functions. It 
is interesting to note the similarity between BI class approximations, as described above, and some fuzzy 
approximators. In the language of fuzzy systems, a basis function is called a rule and an influence function 
is called a membership function. 


BOXES. One of the simplest approximation structures that satisfies Definition 56.4 is the piecewise 
constant function 


h(z;8) = 9° 6T;, 


where 
oo 1, ifzeD,, 
‘10, otherwise, 


where U;Dj = D and DjM D; = G. Often, the Dj are selected in a rectangular grid covering D. In this 
case, very efficient code can be written to calculate the approximation and to update its parameters. 
Generalizations of this concept include using more complex basis elements (e.g., parameterized linear 
functions) in place of the constants 0;, and interpolating between several neighboring Dj; to produce a 
more continuous approximation. A disadvantage is that the influence functions are not continuous. 
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56.3.4 Spline-Like Functions 


Interpolating between regions in the BOXES approach can result in spline functions, such as (for a one 
dimensional domain) 


h(z; 9) = 40; + (1 — 4) Oi41, 


where 


Zj4+1] —Z 
ja coe=s 


= (56.14) 
Zit1 — 2 


for Zj41 > Zz > z and 0; is the value of h(z;®) at z;. When the knots (ie., the z; for i=1,...,N) are 
equally spaced, Equation 56.14 can be rewritten to satisfy Definition 56.4 with constant basis functions 
and influence functions of the form 

lz — zi 


» ifzj1<z< Zi41, 


Tiz={0 Zui —-% (56.15) 


0, otherwise. 


Note that such schemes can be extended to both multidimensional inputs and outputs, at the expense of 
increased memory and computational requirements. For example, a multidimensional output could be 
the state feedback gain as a function of operating condition. 


56.3.5 Radial Basis Functions 


An approximation structure defined as 


h(z,6) = > 002i), 


where I'(z; z;) is a function such as exp(—||z — zill7), which changes with the distance of z from 2, is 
called a Radial Basis Function (RBF) Approximator. In the case where I'(z;z;) has only local support, 
the RBF also satisfies Definition 56.4, with the basis elements being constant functions. Of course, the 
basis functions can again be generalized to be more inclusive functions, at the expense of more involved 
computations. 


56.4 Parameter Estimation 


Section 56.2 has presented various key properties of function approximators and Section 56.3 described 
several neural and other approximation structures that can be used to adaptively approximate unknown 
nonlinearities in uncertain dynamical systems. The present section addresses the problem of designing 
and analyzing parameter estimation algorithms with certain stability and robustness properties. 

Unlike standard parameter estimation problems, in the case of nonlinear dynamical systems with 
unknown nonlinearities, there is a need to first derive a suitable parametric model before designing and 
analyzing the learning scheme. The overall procedure is referred to as adaptive approximation problem, 
which is summarized as follows. 


56.4.1 Adaptive Approximation Problem 


Given an input/output system containing unknown nonlinear functions, the adaptive approximation 
problem deals with the design of online learning schemes and parameter adaptive laws for approximating 
the unknown nonlinearities. 
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The overall design procedure for solving the adaptive approximation problem consists of the following 
three steps: 


1. Derive a parametric model by rewriting the dynamical system in the form 
X(t) = W(S)IF (Z(t) 6%, o*)] +80), (56.16) 


where x(t) € 3t” is a vector that can be computed from available signals, W(s) is a known transfer 
function (in the Laplace s-domain) of dimension n x p, the vector function a. 2H” x HP x MI b> 
i represents an adaptive approximator, z(t) € it” is the input to the adaptive approximator, 6* € 
7° and o* € 7 are unknown “optimal” weights for the adaptive approximator, and 8(t) € R” is 
a possibly filtered version of the MFAE ef (x(t). 

2. Design a learning scheme of the form 


R(t) = L(z(t), A(t), 6(t)), 
e(t) = L(t) — x(t), 


where 6(t), 6(t) are adjustable weights of the adaptive approximator, £ is the structure of the learn- 
ing scheme, and x(t) is an estimate of x(t), which is used to generate the output estimation error 
e(t). The output estimation error e(t) provides a measure of how well the estimator approximates 
the unknown nonlinearities, and therefore is utilized in updating the parameter adaptive laws. 

3. Design a parameter adaptive law for updating A(t) and 6(t), of the form 


A(t) = Ao (z(t), x(8), X(t), (0), 
6(t) = Ag (z(t), x (0), RH), 6(f)), 


where Ag and A, represent the right-hand side of the adaptive law for 6(t) and 6(f)), respectively. 


The role of the filter W(s) will become clear in the subsequent presentation. For some applications, the 
form of the filter W(s) is imposed by the structure of the problem. In other applications, the structure 
of the problem may purposefully be manipulated to insert the filter in order to take advantage of its 
beneficial noise reduction properties. 

The analysis of the learning scheme consists of proving (under reasonable assumptions) the following 
properties: 


Stable Adaptation Property: In the case of zero mismatch error (i.e., 5(t) = 0), the estimation error 
e(t) = X(t) — x(t) remains bounded and asymptotically approaches zero (or a small neighborhood 
of zero). 

Stable Learning Property: In the case of zero mismatch error (i.e., 8(f) = 0), the function approximation 
error f (z(t), A(t), 6(t)) —f((z(t), §*,o*) remains bounded for all z in some domain of interest D 
and asymptotically approaches zero (or is asymptotically less than some threshold € over D). 

Robust Adaptive and Learning Properties: In the case of nonzero mismatch error (i.e., 8(t) 4 0), the 
function approximation error f (z(t), A(t), 6(t)) —f((z(t), 0*,o*), and the estimation error e(t) = 
{(t) — x(t) remain bounded for all z in some domain of interest D and satisfy a small-in-the- 
mean-square property [17] with respect to the magnitude of the mismatch error. 


It is important for the reader to note that the parameter estimation methodologies for neural control and 
adaptive approximation-based control did not develop in a research vacuum, but they are the extension of 
a large number of parameter estimation results. Parametric estimation is a well-established field in science 
and engineering since it is one of the key components in developing models from observations. Several 
books are available for parameter estimation in the context of system identification [21], adaptive control 
[17] and time series analysis [10]. The parameter estimation methods presented herein are based on these 
standard estimation techniques but with special emphasis on the adaptive approximation problem. 
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A significant number of results have been developed for offline parameter estimation, where all the 
data are first collected and then processed to fit an assumed model. Both frequency and time-domain 
approaches can be used, depending on the nature of the input-output data. Moreover, stochastic tech- 
niques have been extensively used to deal with measurement noise and other types of uncertainty. A key 
component in offline parameter estimation is the selection of the norm, which determines the objective 
function to be minimized. 

Adaptive approximation-based control requires the use of online parameter estimation methods; that 
is, techniques that are based on the idea of first choosing an initial estimate for the unknown parameter 
vector, then recursively updating the estimate based on the current set of measurements. One of the 
key characteristics of online parameter estimation methods is that as streaming data become available 
in real-time, it is processed, via updating the of parameter estimates, and then discarded. Therefore, the 
presented techniques require no data storage during real-time processing applications, except possibly 
for some buffering window that can be used to filter measurement noise. In general, the information 
presented by the past history of measurements (in time and/or space) is encapsulated by the current 
value of the parameter estimate. Adaptive parameter estimation methods are used extensively in various 
applications, especially those dealing with time-varying systems or unstable open-loop systems. It is also 
used as a way of avoiding long delays and high costs that result from offline system identification methods. 

As discussed in Section 56.2, adaptive approximators can be classified into two categories of interest: 
linearly parameterized and nonlinearly parameterized. In the case of linearly parameterized approxi- 
mators, the parameters denoted by o are selected a priori and remain fixed. Therefore, the remaining 
adaptable weights ) appear linearly. For nonlinearly parameterized approximators, both 0 and o weights 
are updated online. The case of linearly parameterized approximators allows the derivation of stronger 
analytical results for stability and convergence. 

The adaptive parameter estimation problem can be formulated both in a continuous-time as well as a 
discrete-time framework. In practical applications, the actual plant typically evolves in continuous-time, 
while data processing (parameter estimation, monitoring, etc.) and feedback control is implemented in 
discrete-time using computing devices. Therefore, real-time applications yield so-called hybrid systems, 
where both continuous-time and discrete-time signals are intertwined [1]. 

It is important to keep in mind that different applications may have different objectives relevant to 
parameter convergence. In most control applications that focus on accurate tracking of reference input 
signals, the main objective is not necessarily to make the parameter estimates 0(t) and 6(t) converge to 
the optimal values 0* and o%, respectively, since accurate tracking performance can be achieved without 
convergence of the parameters. In general, parameter convergence is a strong requirement. In applications 
where parameter convergence is desired, the input to the approximator, denoted by z(t), must also satisfy 
a so-called persistency of excitation condition. The structure of the persistence of excitation condition can 
be strongly affected by the choice of function approximator. 


56.5 Nonlinear Control 


Adaptive approximation-based control is based on the concept of designing control algorithms for systems 
with unknown nonlinearities. To understand adaptive approximation-based control methods, it is impor- 
tant to start by considering the simpler problem of designing of nonlinear control algorithms for systems 
with known nonlinearities. In other words, if the nonlinearities were known, how would we go about 
designing a control system? This section provides a general overview of various approaches for nonlinear 
control design. An excellent treatment of nonlinear systems and control methods is given in [18]. 


56.5.1 Small-Signal Linearization and Gain Scheduling 


A traditional approach for dealing with nonlinear systems is to linearize the system around an equilibrium 
point or around a trajectory. This is referred to as small-signal linearization. The main idea behind 
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small-signal linearization is to approximate the nonlinear system by a linear model of the form x = 
Ax + Bu, where A and B are matrices of appropriate dimension. The term small-signal linearization is 
used to characterize the fact that the linear model is an accurate approximation of the nonlinear system 
only in a neighborhood of the point around which the linearization took place. 

A key limitation of the small-signal linearization approach is the fact that the linear model is accurate 
only in a neighborhood around the operating (linearizing) point. To alleviate this limitation, the gain 
scheduling control approach is based on the idea of linearizing around multiple operating points. For each 
linear model there corresponds a feedback controller, thus creating a family of feedback control laws, 
each applicable in the neighborhood of a specific operating point. The family of feedback controllers can 
be combined into a single control whose parameters are changed by a scheduling scheme based on the 
trajectory or some other scheduling variables [34]. 

The performance and robustness of the gain scheduling control approaches is impacted by the fact that 
the controller parameters are precomputed offline for each operating condition. Hence, during operation 
the controller is fixed, even though the linear control gains are changing as the operating conditions 
change. In the presence of modeling errors or changes in the system dynamics, the gain scheduling 
controller may result in deterioration of the performance since the method does not provide any learning 
capability to correct—during operation—any inaccurate schedules. Another possible drawback of the 
gain scheduling approach is that it requires considerable offline effort to derive a reliable gain schedule 
for each possible situation that the plant will encounter. 

The gain scheduling approach can be conveniently viewed as a special case of the adaptive approxima- 
tion approach since a local linear model is an example case of a local approximation function. One of the 
key differences between the standard gain scheduling technique and the adaptive approximation-based 
control approach is the ability of the latter to adjust certain parameters (weights) during operation. Unlike 
gain scheduling, adaptive approximation is designed around the principle of “learning” and thus reduces 
the amount of modeling effort that needs to be expended offline. Moreover, it allows the control scheme 
to deal with unexpected changes in plant dynamics due to faults or severe disturbances. 


56.5.2 Feedback Linearization 


Feedback linearization is one of the most powerful and commonly found techniques in nonlinear control. 
This approach is based on cancelling the nonlinearities by the combined use of feedback and change of 
coordinates. A nonlinear system 

x =f (x)+ G(x)u (56.17) 


is said to be input-state feedback linearizable if there exists a diffeomorphism z = T(x), with T(0) = 0, 
such that 


z=Az+Bp'(z) [u—a(z)], (56.18) 


where (A, B) isa controllable pair and B(z) is an invertible matrix for all z in a domain of interest D, C Kt”. 
Similarly, input-output feedback linearization describes input-output systems that can be linearized by 
the use of feedback. 

Therefore, we see that the class of feedback linearizable systems includes not only systems that can be 
directly transformed to a linear system by feedback, but also systems that can be transformed into that 
form by a nonlinear state transformation. Determining whether a given nonlinear system is feedback 
linearizable and what is an appropriate diffeomorphism are not obvious issues, and in fact they can be 
extremely difficult since in general they involve solving a set of partial differential equations. 

One of the key drawbacks of feedback linearization is that it depends on exact cancellation of non- 
linear functions. If one of the functions is uncertain then cancellation is not possible. This is one of the 
motivations for adaptive approximation-based control. Another possible difficulty with feedback lin- 
earization is that not all systems can be transformed to a linearizable form. Another technique, referred 
to as Backstepping [19], can be applied to a class of systems that may not be feedback linearizable. 
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56.5.3 Robust Nonlinear Control 


The methodologies introduced so far in this section were based on the key assumption that the control 
designer exactly knows the system nonlinearities. In practice, this is not a realistic assumption. Another 
class of nonlinear control design tools are based on the principle of assuming that the unknown compo- 
nents of the nonlinearities are bounded in some way by a known function. If this assumption is satisfied 
then it is possible to derive nonlinear control schemes that utilize these known bounding functions instead 
of the unknown nonlinearities. 

This class of tools is referred to as robust nonlinear control design methods [13,18], and it includes: 
(1) bounding control, (2) sliding mode control, (3) Lyapunov redesign method, (4) nonlinear damping, 
and (5) adaptive bounding. Although these techniques have been extensively studied in the nonlinear 
control literature, they tend to yield conservative control laws, especially in cases where the uncertainty 
is significant. The term “conservative” is used among control engineers to indicate the fact that due to 
the uncertainty the control effort applied is more than needed. As a result, the control signal u(t) may be 
large with rapid temporal variations (due to high-gain feedback). These characteristics may cause several 
problems, such as saturation or excessive wear of the actuators, large error in the presence of measurement 
noise, excitation of unmodeled dynamics, and large transient errors. 

The robust nonlinear control design methods provide an interesting contrast to adaptive 
approximation-based control. Specifically, adaptive approximation-based control can be viewed as a way 
of reducing uncertainty during operation such that the need for conservative robust control can be elimi- 
nated or reduced. Another reason for studying these techniques in the context of adaptive approximation 
is their utilization to guarantee closed-loop stability outside of the approximation region D. 


56.6 Adaptive Approximation-Based Control 


Sections 56.2 and 56.3 have presented approximator properties and structures, respectively. Section 56.4 
discussed methods for parameter estimation and issues related to adaptive approximation, while Section 
56.5 reviewed various nonlinear control design methods. This section brings these different topics together 
in the synthesis of adaptive approximation-based control systems. 

The role of adaptive approximation-based control is to estimate unknown nonlinear functions and can- 
cel their effect using the feedback control signal. Cancelling the estimated nonlinear function allows accu- 
rate tracking to be achieved with a smoother control signal, as compared to the robust nonlinear control 
design methods discussed in the previous section. The trade-off is that the adaptive approximation-based 
controller will typically have much higher state dimension (with the approximator adaptive parameters 
considered as states). This trade-off has become significantly more feasible over the past two decades, 
since controllers are frequently implemented via digital computers that have increased remarkably in 
memory and computational capabilities over this recent time span. 

To illustrate some of the key concepts in the design and analysis of adaptive approximation-based 
control schemes, let us consider a simple scalar system 


x = (fol(x) + f*(x)) + (go(x) +.2* (x))u, (56.19) 


where u(t) is the control input, and y(t) = x(f) is the output. The functions fo(x) and go(x) are the known 
components of the dynamics and f*(x) and g*(x) are the unknown parts of the dynamics. The objective 
is to achieve tracking of a certain signal y(t) by the output y(t). 

The unknown portions of the model will be approximated over the compact region D. This region 
is sometimes referred to as the safe operating envelope. For any system, the region D is physically 
determined at the design stage. For example, an electrical motor is designed to operate within certain 
voltage, current, torque, and speed constraints. If these constraints are violated, then the electrical or 
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mechanical components of the motor may fail; therefore, the controller must ensure that the safe physical 
limits of the system represented by D are not violated. 
The state Equation 56.19 can be expressed as 


t= (fo(x) +f lx 67)) + F(x) + (gol) + B(x 6)) utes (x)u, (56.20) 


where OF and 67 are the unknown “optimal” weight vectors, and & and €; represent the MFAE as defined 
in Equation 56.4. The MFAE is a critical quantity, representing the minimum possible deviation between 
the unknown function f* and the input/output function of the adaptive approximator f (x; 67). In general, 
increasing the number of adjustable weights reduces the MFAE. The UA results indicate that any specified 
approximation accuracy € can be attained uniformly on the compact region D if the number of weights 
is sufficiently large. 

The optimal weight vectors 0; and 07 are unknown quantities required only for analytical purposes. 
Typically, OF (similarly for 07) is defined as the value of 0 that minimizes some cost function as defined 
in Equation 56.2. A special case, for the oo-norm yields the uniform network approximation error over 
all x € D: 


Of := arg min 4 sup 
breRY LxeD 
f 


f(x) —F lx, ip. (56.21) 


The approximation-based feedback linearizing control law is summarized as 


Ua = —Am* + Ja — folx) — by (x) "Oy — vy, (56.22) 

u= a ; (56.23) 
Bo(x) + dg(x)" Oy + Ve 

by =I poy%, for xe D, (56.24) 

0¢ =Ps5 (Fgdgku) , for xED, (56.25) 


where x = x — yq is the tracking error, a», > 0 isa positive design constant, I'y and I's are positive-definite 
matrices, and Ps is the projection operator that will be used to ensure the stabilizability condition on Oi 
The auxiliary terms vy and vg are included to ensure that the state remains within the approximation 
region D. 

In the ideal case of zero MFAE, with vy = vg = 0, the adaptive approximation-based control scheme 
with linearly parameterized approximators guarantees that all the signals remain bounded and the tracking 
error x(t) converges to zero. The stability proof is based on the use of Lyapunov stability theory [27]. 

In the case where the MFAE is nonzero (i.e., the adaptive approximation model is not able to approxi- 
mate exactly the unknown nonlinearities within the region D), it is possible for the parameter estimates 
to become unbounded in their attempt to estimate a function that cannot be physically estimated with 
the current number of weights. To prevent this from happening, the parameter update laws are modified 
by the use of the so-called dead-zone or o-modification algorithms [17]. 

Another issue that needs to be addressed is what happens if the trajectory goes outside the operating 
envelope D, which is a physically defined region over which it is safe and desirable for the system to operate. 
The trajectory generation system ensures that the desired state remains in D. The control designer must 
ensure that the actual state converges to D. Within D the objective is high accuracy trajectory tracking; 
therefore, the designer will select the approximator structure to provide confidence about the capability 
of the approximators f and ¢ to approximate the unknown functions f* and g* accurately for x € D. 

The control algorithms developed earlier had assumed that if x(t) leaves the region D, then the auxiliary 
control terms vf and vg are able to bring the state back within D. Here, we show how to ensure that the 
design of the auxiliary terms vy and vg achieves the objective of bringing the trajectory within D. Let 
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D=R" —D; that is, D is the region outside of D. We assume that outside of D, the unknown functions 
f*(x) and g*(x) are bounded by known nonlinearities as follows: 


filx) <f*(x) <fulx), x€D, 
0 < gi (x) <g*(x) <gu(x), xeED. 


The control design for x € D has already been considered. For x € D, the adaptation of the parameter 
estimates 0, and 0, is stopped and p(x) =0, g(x) = 0; that is, no basis functions are placed in D. 
Therefore, for x € D, the auxiliary terms vy and vg are selected as follows: 


_ Jfu@), ife>=0, 

ss ee ife <0, (56.26) 
_ Jgu(x), ifeug = 0, 

‘r eee if eg <0, (56.27) 


where e(t) = x(t) is the tracking error. The stability of the closed-loop system for x € D is obtained again 
by using Lyapunov stability theory [13]. 

The functions vp and vg are not Lipschitz functions. Their simplicity facilitates a clear discussion of 
methods to enforce convergence to D. Usually these functions are smoothed across the boundary of D 
for practical implementations. 


56.7 Reinforcement Learning 


Optimal control theory was developed formally in the 1950s, following the contributions of L. S. 
Pontryagin and R. Bellman. Pontryagin introduced the minimum principle [28], which gave necessary 
conditions for the existence of optimal trajectories. Bellman introduced the concept of dynamic program- 
ming [7,8] that led to the notion of the celebrated Hamilton—Jacobi-Bellman (HJB) partial differential 
equation, which has the value function V™ as its solution. 

Consider the nth order nonlinear system 


x=f(x)+g(x)u, (56.28) 


where x = [x),...5X,]! © R” is the state and u € R” is the control input. The optimal control problem is 
to find the control signal u to minimize a cost function J(x, u) subject to the constraint on the trajectory 
evolution defined in Equation 56.28. 

When the system model of Equation 56.28 is linear and and the cost function is quadratic in x and u, 
for example: 


a / (xT (x) Qx(x) + [lua)II2) de (56.29) 
to 


where Q is a positive-definite matrix, this becomes the Linear Quadratic Gaussian (LQG) problem [2], 
(i.e., the H2 optimal control problem) and the HJB partial differential equation becomes two separate 
Riccati equations, which can be solved very efficiently. For general nonlinear systems, it is extremely 
difficult to solve the HJB equation to obtain an optimal controller. 
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Iff (x) is nonlinear and known, a standard dynamic programming argument reduces the above optimal 
control problem to finding the value function V* solving the HJB partial differential equation 


1 
eS Z (Vige" vz") +x'Qx=0, (56.30) 


where V;* denotes 0V*/Ox. If there exists a continuously differentiable positive-definite solution V* to 
Equation 56.30, then the optimal controller can be expressed as 


1 
u=—Sg' Vz" (56.31) 


There are two obstacles that complicate the solution to the optimal control problem. The first challenge 
is that it can be extremely difficult to solve the HJB partial differential Equation 56.30 when the nonlinear 
function f(x) is known. The second challenge is that f (x) may be only partially unknown. 

Given the above challenges, many approximate optimal control techniques have been developed for 
nonlinear systems. For example, control approaches such as receding horizon control [23], approximation 
of value functions [6,20,33,37], ADP [9,30,31] have been developed. Many ADP approaches address 
situations where the model is at least partially unknown and therefore address applications similar to 
those discussed in the main body of this chapter. There are numerous varieties of ADP, too many to 
address in this chapter. The interested reader can consult [24,30,35]. As one example of ADP, adaptive 
critic designs (ACD) are based on an algorithm that cycles between a policy-improvement routine (i.e., 
control) and a value-determination (i.e., estimation of V*) operation [5,31]. At each optimizing cycle, 
the algorithm approximates the optimal control law and the value function based on the status of the 
system; therefore, there are at least two function approximation problems being solved during the system 
operation. Theoretically, when system dynamics are time-invariant, as the final time approaches infinity, 
the method yields the optimal control law. 


56.8 Concluding Remarks 


This chapter has discussed techniques for using online function approximation to improve control or state 
estimation performance. Successful implementations require proper selection of an objective function, an 
approximation structure, a training algorithm, and analysis of these three choices under the appropriate 
experimental conditions. 

Several characteristics of function approximators have been discussed. In the literature, much attention 
is devoted to selecting certain function approximators because as a class they have the UA property. By 
itself, this condition is not sufficient. First, the set of function approximators, that are dense on the set 
of continuous functions, is larger than the set of universal function approximators. Second, universal 
function approximation requires that the approximator has an arbitrarily large size (i.e., width). Once 
the size of the approximator is limited, a limitation is also placed on the set of functions that can be 
approximated to within a given e. In a particular application, the designer should instead consider which 
approximation structure can most efficiently represent the class of expected functions. Efficiency should 
be measured both in terms of the number of parameters (i.e., computer memory) and the amount of 
computation (i.e., required computer speed). Due to the inexpensive cost of computer memory and 
the hard constraints on the amount of possible computation due to sampling frequency considerations, 
computational efficiency is often more important in applications. 

Although the goal of many applications presented in the literature is to approximate a given function, 
the approximation accuracy is rarely evaluated directly—even in simulation examples. Instead, perfor- 
mance is often evaluated by analyzing graphs of the training error. This approach can be misleading 
because such graphs indicate only the accuracy of the approximation at the pertinent operating points. 
It is quite possible to have a small sample error over a given time span without ever accurately approxi- 
mating the desired function over the desired region. If the goal is to approximate a function over a given 
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region, then some analysis other than monitoring the sample error during training must be performed 
to determine the success level. In simulation, the approximated and actual functions can be compared 
directly. Successful simulation performance is necessary to provide the confidence for implementation. 
In implementations, where the actual function is not known, a simple demonstration that the function 
has been approximated correctly is to show that the performance does not degrade when the parameter 
update law is turned off. 
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57.1.1 Ten Basic Questions about System Identification 


1. What is system identification? 


System Identification allows you to build mathematical models of a dynamic system based on 


measured data. 


57-1 


57-2 


10. 
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How is that done? 

Essentially by adjusting parameters within a given model until its output coincides as well as 
possible with the measured output. 

How do you know if the model is any good? 

A good test is to take a close look at the model’s output compared to the measurements on a data 
set that was not used for the fit (“Validation Data”). 

Can the quality of the model be tested in other ways? 

It is also valuable to look at what the model could not reproduce in the data (“the residuals”). There 
should be no correlation with other available information, such as the system’s input. 

What models are most common? 

The techniques apply to very general models. Most common models are difference equations 
descriptions, such as ARX and ARMAX models, as well as all types of linear state-space models. 
Lately, black box nonlinear structures, such as Artificial Neural Networks, Fuzzy models, and so 
on, have been much used. 

Do you have to assume a model of a particular type? 

For parametric models, you have to specify the structure. However, if you just assume that the 
system is linear, you can directly estimate its impulse or step response using Correlation Analysis 
or its frequency response using Spectral Analysis. This allows useful comparisons with other 
estimated models. 

How do you know what model structure to try? 

Well, you do not. For real life systems there is never any “true model,” anyway. You have to be 
generous at this point, and try out several different structures. 

Can nonlinear models be used in an easy fashion? 

Yes. Most common model nonlinearities are such that the measured data should be nonlinearly 
transformed (like squaring a voltage input if you think that it is the power i-e., the stimulus). Use 
physical insight about the system you are modeling and try out such transformations on models 
that are linear in the new variables, and you will cover a lot. 

What does this article cover? 

After reviewing an archetypical situation in this section, we describe the basic techniques for 
parameter estimation in arbitrary model structures. Section 57.3 deals with linear models of black 
box structure, and Section 57.5 deals with particular estimation methods that can be used (in addi- 
tion to the general ones) for such models. Physically parameterized model structures are described 
in Section 57.6, and nonlinear black box models (including neural networks) are discussed in 
Section 57.7. The final Section 57.8 deals with the choices and decisions the user is faced with. 

Is this really all there is to system identification? 

Actually, there is a huge amount written on the subject. Experience with real data is the driving 
force to further understanding. It is important to remember that any estimated model, no matter 
how good it looks on your screen, has only picked up a simple reflection of reality. Surprisingly 
often, however, this is sufficient for rational decision making. 


57.1.2 Background and Literature 


System identification has its roots in standard statistical techniques and many of the basic routines have 
direct interpretations as well known statistical methods such as least squares and maximum likelihood. 
The control community took an active part in the development and application of these basic techniques to 
dynamic systems right after the birth of “modern control theory” in the early 1960s. Maximum likelihood 
estimation was applied to difference equations (ARMAX models) by Astrém and Bohlin (1965) and 
thereafter a wide range of estimation techniques and model parameterizations flourished. By now, the 
area is well matured with established and well-understood techniques. Industrial use and application of 
the techniques has become standard. See Ljung (2007) for a common software package. 
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The literature on system identification is extensive. For a practical user oriented introduction we may 
mention Ljung and Glad (1994). Texts that go deeper into the theory and algorithms include Ljung (1999), 
and Séderstrém and Stoica (1989). A classical treatment is Box and Jenkins (1970). 

These books all deal with the “mainstream” approach to system identification, as described in this 
article. In addition, there is a substantial literature on other approaches, such as “set membership” (com- 
pute all those models that reproduce the observed data within a certain given error bound), estimation 
of models from given frequency response measurement (Pintelon and Schoukens, 2001), online model 
estimation (Ljung and Séderstrém, 1983), nonparametric frequency domain methods (Brillinger, 1981), 
gray-box identification (Bohlin, 2006) and so on. To follow the development in the field, the IFAC series 
of Symposia on System Identification (Saint Malo, 2009; Newcastle, 2006) is a good source. A recent 
perspective paper is Ljung (2010). 


57.1.3 An Archetypical Problem: ARX Models and the Linear 
Least-Squares Method 


57.1.3.1 The Model 


We shall generally denote the system’s input and output at time t by u(t) and y(t), respectively. Perhaps, 
the most basic relationship between the input and output is the linear difference equation 


y(t) +ayy(t — 1) +--+ +any(t — 1) = bu(t — 1) +---+ by, u(t — m) (57.1) 


We have chosen to represent the system in discrete time, primarily since observed data are always collected 
by sampling. It is thus more straightforward to relate observed data to discrete time models. Nothing 
prevents us however from working with continuous time models: we shall return to that in Section 57.6. 
In Equation 57.1, we assume the sampling interval to be one time unit. This is not essential, but makes 
notation easier. 
A pragmatic and useful way to see Equation 57.1 is to view it as a way of determining the next output 
value given previous observations: 


y(t) = —ayy(t — 1) —---—any(t —n) + by u(t — 1) +--+ + bmu(t — m) (57.2) 
For more compact notation we introduce the vectors 


0=[ay,..-54n b1,---50m]! (57.3) 
o(t) = [—y(t—1)---—y(t—n) u(t —1)--- u(t —m)]? (57.4) 


With these Equation 57.2 can be rewritten as 
y(t) = 9"(1)6 


To emphasize that the calculation of y(t) from past data Equation 57.2 indeed depends on the parameters 
in 0, we shall rather call this calculated value p(t|9) and write 


j(t|0) = 7 (£)0 (57.5) 


57.1.3.2 The Least-Squares Method 


Now suppose for a given system that we do not know the values of the parameters in 0, but that we have 
recorded inputs and outputs over a time interval 1 < t < N: 


ZN = {u(1), y(),.-.,u(N), (N)} (57.6) 


57-4 Control System Advanced Methods 


An obvious approach is then to select 6 in Equations 57.1 through 57.5 so as to fit the calculated values 
¥(t|9) as well as possible to the measured outputs by the least-squares method: 


min Vn (0, ZN) (57.7) 


where 


N 
Vn (0, ZY ae (y(t) — $(t|9))? 
aah (57.8) 


N 
w 00 — 9" (16) 


We shall denote the value of @ that minimizes Equation 57.7 by 6: 


Oy = arg min Vn (8, ZN) (57.9) 


(“arg min” means the minimizing argument, i.e., that value of @ which minimizes Vy.) 
Since Vy is quadratic in 0, we can find the minimum value easily by setting the derivative to zero: 


d 2 & 
0= qo WN O ZN) = a » o(t)(y(t) — o! (t)8) 


which gives 


N N 
> oyt) = > (to (10 (57.10) 
t=1 tI 

or 

na N N 
on = bs How) > o(t)y(t) (57.11) 
t=1 t=1 


Once the vectors ¢(f) are defined, the solution can easily be found by modern numerical software, such 
as MATLAB®, 


Example 57.1: First-Order Difference Equation 
Consider the simple model 
y(t) + ay(t — 1) = bu(t — 1). 
This gives us the estimate according to Equations 57.3, 57.4, and 57.11 
5 -1 
an] _[oy7t—1) — Dy(t—1ult—-1) Y y(dy(t- 1) 
by —Siy(t—1ult—1)  Sou2(t—1) > y(t)u(t — 1) 


All sums are from t = 1 to t= N. A typical convention is to take values outside the measured range 
to be zero. In this case we would thus take y(0) = 0. 


The simple model (Equation 57.1) and the well-known least-squares method (Equation 57.11) form 
the archetype of System Identification. Not only that—they also give the most commonly used parametric 
identification method and are much more versatile than perhaps perceived at first sight. In particular, 
one should realize that Equation 57.1 can directly be extended to several different inputs (this just calls 
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for a redefinition of g(t) in Equation 57.4) and that the inputs and outputs do not have to be the raw 
measurements. On the contrary—it is often most important to think over the physics of the application 
and come up with suitable inputs and outputs for Equation 57.1, formed from the actual measurements. 


Example 57.2: An Immersion Heater 


Consider a process consisting of an immersion heater immersed in a cooling liquid. We measure: 


+ v(t): The voltage applied to the heater 
+ r(t): The temperature of the liquid 
+ y(t): The temperature of the heater coil surface 


Suppose we need a model for how y(t) depends on r(t) and v(t). Some simple considerations based 
on common sense and high school physics (“Semi-physical modeling”) reveal the following: 


+ The change in temperature of the heater coil over one sample is proportional to the electrical 
power in it (the inflow power) minus the heat loss to the liquid 

+ The electrical power is proportional to v2(t) 

+ The heat loss is proportional to y(t) — r(t) 


This suggests the model 
y(t) = y(t = 1) + av? (t= 1) — B(t = 1) = r(t = 1) 


which fits into the form 
y(t) + @1y(t — 1) = G2v2(t — 1) + 3r(t — 1) 


This is a two input (v2 and r) and one output model, and corresponds to choosing 
e(t)=[-yltt-1) v(t—1) r(t- 1)" 


in Equation 57.5. 


57.1.3.3 Some Statistical Remarks 


Model structures, such as Equation 57.5 that are linear in 6 are known in statistics as linear regression 
and the vector g(t) is called the regression vector (its components are the regressors). “Regress” here 
alludes to the fact that we try to calculate (or describe) y(t) by “going back” to y(t). Models such as 
Equation 57.1 where the regression vector—(t)—contains old values of the variable to be explained— 
y(t)—are then partly auto-regressions. For that reason the model structure Equation 57.1 has the standard 
name ARX-model (Auto-regression with extra inputs). 

There is a rich statistical literature on the properties of the estimate Oy under varying assumptions. 
See, for example, Draper and Smith (1981). So far we have just viewed Equations 57.7 and 57.8 as “curve- 
fitting.” In Section 57.2.2, we shall deal with a more comprehensive statistical discussion, which includes 
the ARX model as a special case. Some direct calculations will be done in the following subsection. 


57.1.3.4 Model Quality and Experiment Design 


Let us consider the simplest special case, that of a Finite Impulse Response (FIR) model. That is obtained 
from Equation 57.1 by taking n = 0: 


y(t) = byu(t — 1) +-+-+ bmu(t — m) (57.12) 
Suppose that the observed data really have been generated by a similar mechanism 
y(t) = Blut — 1) +--+ + bP u(t — m) + e(t) (57.13) 


where e(t) is a white noise sequence with variance i, but otherwise unknown. (i.e., e(t) can be described 
as a sequence of independent random variables with zero mean values and variances 4.) Analogous to 
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Equation 57.5, we can write this as 
y(t) = 9" (t)% + e(t) (57.14) 


We can now replace y(t) in Equation 57.11 by the above expression, and obtain 


i N “lyn 
Oy = bs How" Y= ot)y(t) 
t=1 


t=1 


N to N 
= pa How") p» g(t)" (t)®0+ >> aoet| 


t=1 t=1 t=1 


or 


N “lyn 
On = 6y — 0 = bs “ow Yo eet) (57.15) 
t=1 


t=1 


Suppose that the input u is independent of the noise e. Then ¢ and e are independent in this expression, 
so it is easy to see that Ey = 0, since e has zero mean. The estimate is consequently unbiased. Here E 
denotes mathematical expectation. 

We can also form the expectation of 602, that is, the covariance matrix of the parameter error. 
Denote the matrix within brackets by Ry. Take expectation with respect to the white noise e. Then Ry is 
a deterministic matrix and we have 


N 
Py = E0nOK =Ry' D> g(t)" (s)Ee(t)e(s)Ry' = Ry! (57.16) 


t,s=1 


since the double sum collapses to \Ry. 
We have thus computed the covariance matrix of the estimate 9y,. It is determined entirely by the input 
properties and the noise level. Moreover, define 


‘ 1 
R= lim —Ry (57.17) 


This will be the covariance matrix of the input, that is, the i—j-element of R is Ry,(i—j), as defined by 
Equation 57.104 later on. 

If the matrix R is nonsingular, we find that the covariance matrix of the parameter estimate is approx- 
imately (and the approximation improves as N — oo) 


a= 
Py = —R7! 57.18 
NS ( ) 


A number of things follow from this. All of them are typical of the general properties to be described in 
Section 57.2.2: 


* The covariance decays like 1/N, so the parameters approach the limiting value at the rate 1//N. 

¢ The covariance is proportional to the noise-to-signal ratio. That is, it is proportional to the noise 
variance and inversely proportional to the input power. 

¢ The covariance does not depend on the input’s or noise’s signal shapes, only on their vari- 
ance/covariance properties. 


l« 


« Experiment design, that is, the selection of the input u, aims at making the matrix R~! “as small as 


possible.” Note that the same R can be obtained for many different signals u. 
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57.1.4 The Main Ingredients 


The main ingredients for the System Identification problem are as follows: 


¢ The data set ZN 

« Aclass of candidate model descriptions; a Model Structure 
¢ Accriterion of fit between data and models 

¢ Routines to validate and accept resulting models 


We have seen in Section 57.1.3, a particular model structure, the ARX-model. In fact the major prob- 
lem in system identification is to select a good model structure, and a substantial part of this article 
deals with various model structures. See Sections 57.3, 57.6, and 57.7, which all concern this problem. 
Generally speaking, a model structure is a parameterized mapping from past inputs and outputs Z‘~! (cf 
Equation 57.6) to the space of the model outputs: 


7(t|6) = g(0,Z'1) (57.19) 


Here 0 is the finite dimensional vector used to parameterize the mapping. 

Actually, the problem of fitting a given model structure to measured data is much simpler, and can be 
dealt with independently of the model structure used. We shall do so in the following section. 

The problem of assuring a data set with adequate information contents is the problem of experiment 
design, and it will be described in Section 57.8.1. 

Model validation is both a process to discriminate between various model structures and the final 
quality control station, before a model is delivered to the user. This problem is discussed in Section 57.8.2. 


57.2 General Parameter Estimation Techniques 


In this section, we shall deal with issues that are independent of model structure. Principles and algorithms 
for fitting models to data, as well as the general properties of the estimated models are all model-structure 
independent and equally well applicable to, say, ARMAX models and Neural Network models. 

The section is organized as follows. In Section 57.2.1, the general principles for parameter estimation are 
outlined. Sections 57.2.2 and 57.2.3 deal with the asymptotic (in the number of observed data) properties 
of the models, while algorithms, both for online and off-line use are described in Section 57.2.5. 


57.2.1 Fitting Models to Data 


In Section 57.1.3, we showed one way to parameterize descriptions of dynamical systems. There are many 
other possibilities and we shall spend a fair amount of this contribution to discuss the different choices 
and approaches. This is actually the key problem in system identification. No matter how the problem is 
approached, the bottom line is that such a model parameterization leads to a predictor 


(t|0) = g(0,Z'') (57.20) 


that depends on the unknown parameter vector and past data Z'~! (see Equation 57.6). This predictor 
can be linear in y and u. This in turn contains several special cases both in terms of black box models and 
physically parameterized ones, as will be discussed in Sections 57.3 and 57.6, respectively. The predictor 
could also be of general, nonlinear nature, as discussed in Section 57.7. 

In any case we now need a method to determine a good value of 9, based on the information in an 
observed, sampled data set (Equation 57.6). It suggests itself that the basic least-squares like approach 
Equations 57.7 through 57.9 still is a natural approach, even when the predictor j(t|®) is a more general 
function of 0. 
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A procedure with some more degrees of freedom is the following one 
1. From observed data and the predictor (t|9) form the sequence of prediction errors, 
e(t, 0) = y(t) — p(t|9),  f=1,2,...N (57.21) 
2. Possibly filter the prediction errors through a linear filter L(q), 
ep(t, 0) = L(q)e(t, 8) (57.22) 


(here q denotes the shift operator, qu(t) = u(t+1)) so as to enhance or depress interesting or 
unimportant frequency bands in the signals. 
3. Choosea scalar valued, positive function £(-) so as to measure the “size” or “norm” of the prediction 


error: 

L(epr(t, 9)) (57.23) 
4. Minimize the sum of these norms: 

a = arg min Vn (0, ZN) (57.24) 

where 

ia 
Ny _ 

Vw@2Z%) == 2 Mert 9) (57.25) 


This procedure is natural and pragmatic—we can still think of it as “curve-fitting” between y(t) and 
y(t|9). It also has several statistical and information theoretic interpretations. Most importantly, if the 
noise source in the system (like in Equation 57.61 below) is supposed to be a sequence of independent 
random variables {e(t)} each having a probability density function f(x), then Equation 57.24 becomes 
the maximum likelihood estimate (MLE) if we choose 


L(q)=1 and &e) = —logfe(e) (57.26) 


The MLE has several nice statistical features and thus gives a strong “moral support” for using the 
outlined method. Another pleasing aspect is that the method is independent of the particular model 
parameterization used (although this will affect the actual minimization procedure). For example, the 
method of “back propagation” often used in connection with neural network parameterizations amounts 
to computing 6 in Equation 57.24 by a recursive gradient method. We shall deal with these aspects in 
Section 57.2.5. 


57.2.2 Model Quality 


An essential question is, of course, what properties will the estimate have resulting from Equation 57.24. 
These will naturally depend on the properties of the data record ZN defined by Equation 57.6. It is in 
general a difficult problem to characterize the quality of On exactly. One normally has to be content with 
the asymptotic properties of Oy as the number of data, N, tends to infinity. 
It is an important aspect of the general identification method Equation 57.24 that the asymptotic prop- 
erties of the resulting estimate can be expressed in general terms for arbitrary model parameterizations. 
The first basic result is the following one: 


6y > 6* asN—oo where (57.27) 
6* = arg min Ef(er(t, 9)) (57.28) 


That is, as more and more data become available, the estimate converges to that value 6*, that would 
minimize the expected value of the “norm” of the filtered prediction errors. This is, in a sense the best 
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possible approximation of the true system that is available within the model structure. The expecta- 
tion E in Equation 57.28 is taken with respect to all random disturbances that affect the data and it 
also includes averaging over the input properties. This means in particular that 6* will make p(t|@*) a 
good approximation of y(t) with respect to those aspects of the system that are enhanced by the input 
signal used. 

The second basic result is the following one: If {e(t, 8*)} is approximately white noise, then the covari- 
ance matrix of 6 is approximately given by 


By — 9° )y — 07)" ~ Steweoy (or (57.29) 

where 
h = Ee*(t, 0*) (57.30) 
0) = S501) a (5731) 


Think of tp as the sensitivity derivative of the predictor with respect to the parameters. Then Equation 57.29 
says that the covariance matrix for 6y is proportional to the inverse of the covariance matrix of this 
sensitivity derivative. This is a quite natural result. 

Note: For all these results, the expectation operator E can, under general conditions, be replaced by the 
limit of the sample mean, that is 


N 
EWYT() > Jim — Y weyr) (57.32) 
t=1 


The results, Equations 57.27 through 57.31 are general and hold for all model structures, both linear 
and nonlinear ones, subject only to some regularity and smoothness conditions. They are also fairly 
natural, and will give the guidelines for all user choices involved in the process of identification. See Ljung 
(1999) for more details around this. 


57.2.3 Measures of Model Fit 


Some quite general expressions for the expected model fit, that are independent of the model structure, 
can also be developed. 
Let us measure the (average) fit between any model (Equation 57.20) and the true system as 


V (0) = Ely(t) — pel)? (57.33) 


Here expectation E is over the data properties (i-e., expectation over “Z°°” with the notation Equa- 
tion 57.6). Recall that expectation also can be interpreted as sample means as in Equation 57.32. 

Before we continue, let us note the very important aspect that the fit V will depend, not only on the 
model and the true system, but also on data properties, like input spectra, possible feedback, and so on. 
We shall say that the fit depends on the experimental conditions. 

The estimated model parameter 6y is a random variable, because it is constructed from observed data, 
that can be described as random variables. To evaluate the model fit, we then take the expectation of 
vi Nv) with respect to the estimation data. That gives our measure 


Fy = EV(6n) (57.34) 
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In general, the measure Fy depends on a number of things: 


¢ The model structure used 

¢ The number of data points N 

« The data properties for which the fit V is defined 
¢ The properties of the data used to estimate Oy 


The rather remarkable fact is that if the two last data properties coincide, then, asymptotically in N [see, 
e.g., Ljung (1999), Chapter 16] 


: dimo 
Fy © Vy(6*) (1+ ~ ) (57.35) 


Here 6* is the value that minimizes the expected criterion (Equation 57.28). The notation dim ®@ means 
the number of estimated parameters. The result also assumes that the criterion function ¢(¢) = ||e|| 2 and 
that the model structure is successful in the sense that ep(f) is approximately white noise. 

Despite the reservations about the formal validity of Equation 57.35, it carries a most important 
conceptual message: If a model is evaluated on a data set with the same properties as the estimation data, 
then the fit will not depend on the data properties, and it will depend on the model structure only in terms 
of the number of parameters used and of the best fit offered within the structure. 

The expression can be rewritten as follows. Let yo(t|t — 1) denote the “true” one step ahead prediction 
of y(t), and let 


W(0) = Elpo(tlt — 1) — p(e|®)|* (57.36) 


and let 
h = Ely(t) — fo(tlt — DI? (57.37) 


Then X is the innovations variance, that is, that part of y(t) that cannot be predicted from the past. 
Moreover, W(6*) is the bias error, that is, the discrepancy between the true predictor and the best one 
available in the model structure. Under the same assumptions as above, Equation 57.35 can be rewritten 
as 
dim 8 
N 


Fy ~X+W(6*) +2 (57.38) 


The three terms constituting the model error then have the following interpretations: 


¢ is the unavoidable error, stemming from the fact that the output cannot be exactly predicted, 
even with perfect system knowledge. 

« W(0*) is the bias error. It depends on the model structure, and on the experimental conditions. It 
will typically decrease as dim 0 increases. 

¢ The last term is the variance error. It is proportional to the number of estimated parameters and 
inversely proportional to the number of data points. It does not depend on the particular model 
structure or the experimental conditions (as long as these are the same for the estimation and 
evaluation). 


57.2.4 Model Structure Selection 


The most difficult choice for the user is no doubt to find a suitable model structure to fit the data to. 
This is of course a very application-dependent problem, and it is difficult to give general guidelines. (Still, 
some general practical advice will be given in Section 57.8.) 

At the heart of the model structure selection process is to handle the trade-off between bias and 
variance, as formalized by Equation 57.38. The “best” model structure is the one that minimizes Fy, the 
fit between the model and the data for a fresh data set—one that was not used for estimating the model. 
Most procedures for choosing the model structures are also aiming at finding this best choice. 
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57.2.4.1 Cross Validation 


A very natural and pragmatic approach is cross validation. This means that the available data set is split 
into two parts, estimation data, Uh st that is used to estimate the models: 


On, = arg min Vy, (6, Ze) (57.39) 
and validation data, Ze for which the criterion is evaluated: 
Ey, = = Vy, (6n,,Z a (57.40) 


Here Vy is the criterion Equation 57.25. Then Fy will be an unbiased estimate of the measure Fy, defined 
by (Equation 57.34), which was discussed at length in the previous section. The procedure would the be 
to try out a number of model structures, and choose the one that minimizes Fy, . 

Such cross validation techniques to find a good model structure has an immediate intuitive appeal. 
We simply check if the candidate model is capable of “reproducing” data have it has not yet seen. If that 
works well, we have some confidence in the model, regardless of any probabilistic framework that might 
be imposed. Such techniques are also the most commonly used ones. 

A few comments could be added. In the first place, one could use different splits of the original data into 
estimation and validation data. For example, in statistics, there is a common cross validation technique 
called “leave one out.” This means that the validation data set consists of one data point “at a time,” but 
successively applied to the whole original set. In the second place, the test of the model on the validation 
data does not have to be in terms of the particular criterion (Equation 57.40). In system identification it 
is common practice to simulate (or predict several steps ahead) the model using the validation data, and 
then visually inspect the agreement between measured and simulated (predicted) output. 


57.2.4.2 Estimating the Variance Contribution: Penalizing the Model Complexity 


It is clear that the criterion (Equation 57.40) has to be evaluated on the validation data to be of any use—it 
would be strictly decreasing as a function of model flexibility if evaluated on the estimation data. In other 
words, the adverse effect of the dimension of 8 shown in Equation 57.38 would be missed. There are a 
number of criteria—often derived from entirely different viewpoints—that try to capture the influence 
of this variance error term. The two best-known ones are Akaike’s information theoretic criterion (AIC), 
which has the form (for Gaussian disturbances) 


Vn (0, ZN) = (1+ es : aa ry (57.41) 


and Rissanen’s minimum description length criterion (MDL) in which dim @ in the expression above is 
replaced by log N dim 9. See Akaike (1974a) and Rissanen (1978). 

The criterion Vy is then to be minimized both with respect to 0 and to a family of model structures. 
The relation to the expression (Equation 57.35) for Fy is obvious. 


57.2.5 Algorithmic Aspects 


In this section, we discuss how to achieve the best fit between observed data and the model, that is, how 
to carry out the minimization of Equation 57.24. For simplicity, we here assume a quadratic criterion and 
set the prefilter L to unity: 


Vy (0) = ay Iy(t) — 5(t19)/? (57.42) 


No analytic solution to this problem is possible ee the model 9(t|6) is linear in 9, so the minimization 
has to be done by some numerical search procedure. A classical treatment of the problem of how to 
minimize the sum of squares is given in Dennis and Schnabel (1983). 
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Most efficient search routines are based on iterative local search in a “downhill” direction from the 
current point. We then have an iterative scheme of the following kind 


AFD — GO — WRONG, (57.43) 
Here 6 is the parameter estimate after iteration number i. The search scheme is thus made up of the 
three entities 


e Wj step size 
+g; an estimate of the gradient V,, (0) 
e R;amatrix that modifies the search direction 


It is useful to distinguish between two different minimization situations 


i. Off-line or batch: The update wiR; | g} is based on the whole available data record ZY. 
ii. Online or recursive: The update is based only on data up to sample i (Z'), (typically done so that 
the gradient estimate g; is based only on data just before sample i). 


We discuss these two modes separately below. First, some general aspects will be treated. 


57.2.5.1 Search Directions 
The basis for the local search is the gradient 


dVn (0 a 
V4,(0) = EO) _ Sot) —FeI)WUL.) (57.44) 
t=1 
where 3 
H(t, 6) = SH (t10) (57.45) 


The gradient 1 is in the general case a matrix with dim 0 rows and dim y columns. It is well known that 
gradient search for the minimum is inefficient, especially close to the minimum. Then it is optimal to use 
the Newton search direction 


R710) Vx,(0) (57.46) 
where 
m d?Vn (0) 
R(0) = Viv(0) = — 
ieee cS oP 
= 7 VE OYTO) + — DOW — Hit) Se HEP) (57.47) 


t=1 t=1 
The true Newton direction will thus require that the second derivative 
oO, 
ez tl®) 
be computed. Also, far from the minimum, R(9) need not be positive semidefinite. Therefore, alternative 


search directions are more common in practice: 


¢ Gradient direction: Simply take 
R=I (57.48) 


¢ Gauss—Newton direction: Use 


N 
1 An fa 
R=Hi= 5 Y(t, 6 y(t, 6) (57.49) 
t=1 
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« Levenberg-Marquard direction: Use 


R; =H;+ 81 


where Hj is defined by Equation 57.49. 

¢ Conjugate gradient direction: Construct the Newton direction from a sequence of gradient estimates. 
Loosely, think of Vx, as constructed by difference approximation of d gradients. The direction in 
Equation 57.46 is however, constructed directly, without explicitly forming and inverting V”’. 


It is generally considered (Dennis and Schnabel, 1983), that the Gauss—Newton search direction is to 
be preferred. For ill-conditioned problems, the Levenberg-Marquard modification is recommended. 


57.2.5.2 Online Algorithms 


The expressions (Equations 57.44 and 57.47) for the Gauss-Newton search clearly assume that the whole 
data set ZN is available during the iterations. If the application is of an off-line character, that is, the model 
&v is not required during the data acquisition, this is also the most natural approach. 

However, many adaptive situations require online (or recursive) algorithms, where the data are pro- 
cessed as they are measured. (Such algorithms are in Neural Network contexts often also used in off-line 
situations. Then the measured data record is concatenated with itself several times to create a (very) long 
record that is fed into the online algorithm.) We may refer to Ljung and Sdderstrém (1983) as a basic 
source of information for recursive identification techniques. 

It is natural to consider the following algorithm as the basic one: 


6(t) = 6(t — 1) + wp RP Wt, A(t — 1))e(t, A(t — 1) (57.50) 
e(t, 0) = y(t) — 9(t16) (57.51) 
Ry = Rea + wel h(t, 0(¢ — D7 (t, 6(¢ — 1)) — Rr] (57.52) 


The reason is that if j(¢|9) is linear in 0, then Equations 57.50 through 57.52, with ; = 1/t, provides 
the analytical solution to the minimization problem Equation 57.42. This also means that this is a natural 
algorithm close to the minimum, where a second order expansion of the criterion is a good approximation. 
In fact, it is shown in Ljung and Sdderstrém (1983), that Equations 57.50 through 57.52 in general gives 
an estimate 4(t) with the same (“optimal”) statistical, asymptotic properties as the true minimum to 
Equation 57.42. 

It should be mentioned that the quantities H(t|A(t —1)) and W(t, A(t —1)) would normally (except 
in the linear regression case) require the whole data record to be computed. This would violate the 
recursiveness of the algorithm. In practical implementations these quantities are therefore replaced by 
recursively computed approximations. The idea behind these approximations is to use the defining 
equation for (t|®) and y(t, 8) (which typically are recursive equations), and replace any appearance of 0 
with its latest available estimate. See Ljung and Séderstrém (1983) for more details. 

Some averaged variants of Equations 57.50 through 57.52 have also been discussed: 


6(t) = 6(t — 1) + wp Rp W(t, A(t — 1))e(t, A(t — 1) (57.53) 
6(t) = A(t — 1) + p/[6(t) — O(t — )] (57.54) 
The basic algorithm of Equations 57.50 through 57.52 then corresponds to p; = 1. Using ¢; < 1 gives a 


so called “accelerated convergence” algorithm. It was introduced by Polyak and Juditsky (1992) and has 
then been extensively discussed by Kushner and Yang (1993), and others. The remarkable thing with 
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this averaging is that we achieve the same asymptotic statistical properties of A(t) by Equations 57.53 and 
57.54 with R; = I (gradient search) as by Equations 57.50 through 57.52 if 


Or=1/t we>>pr wr 


It is thus an interesting alternative to Equations 57.50 through 57.52, in particular if dim 6 is large so R; 
is a big matrix. 


57.2.5.3 Local Minima 


A fundamental problem with minimization tasks like Equation 57.42 is that Vy (8) may have several or 
many local (nonglobal) minima, where local search algorithms may get caught. There is no easy solution 
to this problem. It is usually well worth the effort to find a good initial value 6© where to start the 
iterations. Other than that, only various global search strategies are left, such as random search, random 
restarts, simulated annealing, and the genetic algorithm. 


57.3 Linear Black Box Systems 


57.3.1 Linear System Descriptions in General 


57.3.1.1 A Linear System with Additive Disturbances 


A linear system with additive disturbances v(t) can be described by 
y(t) = G(q)u(t) + v(t) (57.55) 


Here u(t) is the input signal, and G(q) is the transfer function from input to output y(t). The symbol q is 
the shift operator, so Equation 57.55 should be interpreted as 


y(t) = D> geu(t — k) + v(t) = (> ar) u(t) + v(t) (57.56) 
k=0 k=0 


The disturbance v(t) can in general terms be characterized by its spectrum, which is a description of 
its frequency content. It is often more convenient to describe v(t) as being (thought of as) obtained by 
filtering a white noise source e(t) through a linear filter H(q): 


v(t) = H(q)e(t) (57.57) 
This is, from a linear identification perspective, equivalent to describing v(t) as a signal with spectrum 
®,(w) = r|H(e)|? (57.58) 


where } is the variance of the noise source e(t). We shall assume that H(q) is normalized to be monic, 
that is, 


CO 
H(q)=1+ >> hq * (57.59) 
k=1 


Putting all of this together, we arrive at the standard linear system description 


y(t) = G(q)u(t) + H(qg)e(t) (57.60) 
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57.3.1.2 Parameterized Linear Models 


Now, if the transfer functions G and H in Equation 57.60 are not known, we would introduce parameters 
8 in their description that reflect our lack of knowledge. The exact way of doing this is the topic of this 
section as well as of Section 57.6. 

In any case the resulting, parameterized model will be described as 


y(t) = G(q, 9)u(t) + H(q, ®)e(t) (57.61) 


The parameters 6 can then be estimated from data using the general procedures described in Section 57.2. 


57.3.1.3 Predictors for Linear Models 


Given a system description of Equation 57.61 and input-output data up to time ft — 1, 
y(s), u(s) s<t—1 (57.62) 
how shall we predict the next output value y(t)? 


In the general case of Equation 57.61 the prediction can be deduced in the following way: Divide 
Equation 57.61 by H(q, 9): 


H~'(q,)y(t) = H~'(q, 0)G(q, ®)u(t) + e(t) 


or 


y(t) = [1 — H~*(q, 8) y(t) + Hq, 8) G(q, ®)u(t) + e(t) (57.63) 


In view of the normalization of Equation 57.59 we find that 


A(q, 9) —1 —k 
= h 
H@9) — H@,9) » - 


H~'(q,0) = 


The expression [1 — H~!(q,®)]y(t) thus only contains old values of y(s), s< t—1. The right side of 
Equation 57.63 is thus known at time t — 1, with the exception of e(t). The prediction of y(t) is simply 
obtained from Equation 57.63 by deleting e(t): 


$(t|0) = [1 —H~'(q, 9) y(t) + Hq, 0)G(q, ®)u(t) (57.64) 


This is a general expression for how ready-made models predict the next value of the output, given old 
values of y and u. Inserting the predictor in Equation 57.25 (with ¢(e) = &*) give the identification criterion 


N 


ra 1g, (y(t) — G(q, u(t) (57.65) 


57.3.1.4 A Characterization of the Limiting Model in a General Class of Linear Models 


Let us apply the general limit result in Equations 57.27 and 57.28 to the linear model structure (Equa- 
tion 57.61 or 57.64). If we choose a quadratic criterion ¢(¢) = ¢? (in the scalar output case) then this result 
tells us, in the time domain, that the limiting parameter estimate is the one that minimizes the filtered 
prediction error variance (for the input used during the experiment). Suppose that the data actually have 
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been generated by 
y(t) = Go(q)u(t) + v(t) (57.66) 


Let ®,,(w) be the input spectrum and ®,(w) be the spectrum for the additive disturbance v. Then the 
filtered prediction error can be written 


ep(t,0) = Oy) — Gq. ult) 
ae ) (57.67) 
=5 a py l(Gola) — GCG, O))ult) + (8) 


By Parseval’s relation, the prediction error variance can also be written as an integral over the spectrum 
of the prediction error. This spectrum, in turn, is directly obtained from Equation 57.67, so the limit 
estimate 6* in Equation 57.28 can also be defined as 


Tp, wo gn 2 ul@)|L(e)/? 
= iw 10) 2 
6* = arg min | [cote )— G(e™, 8)| Hee 7° 


Tv 
+ i. ,(w)|L(e)? /|H(e®, eyPao (57.68) 
—Tt 
If the noise model H(q, 9) = H,.(q) does not depend on 0 [as in the output error model (Equation 57.75)] 
the expression (Equation 57.68) thus shows that the resulting model G(e“”, 6*) will give that frequency 
function in the model set that is closest to the true one, in a quadratic frequency norm with weighting 

function 
Q(w) = &,(w)|L(e)?/|H(e") |? (57.69) 


This shows clearly that the fit can be affected by the choice of prefilter L, the input spectrum ®, and the 
noise model H,.. 


57.3.2 Linear and Ready-Made Models 


Sometimes we are faced with systems or subsystems that cannot be modeled based on physical insights. 
The reason may be that the function of the system or its construction is unknown or that it would be 
too complicated to sort out the physical relationships. It is then possible to use standard models, which 
by experience are known to be able to handle a wide range of different system dynamics. Linear systems 
constitute the most common class of such standard models. From a modeling point of view these models 
thus serve as ready-made models: tell us the size (model order), and it should be possible to find something 
that fits (to data). 


57.3.2.1 A Family of Transfer Function Models 


A very natural approach is to describe G and H in Equation 57.61 as rational transfer functions in the 
shift (delay) operator with unknown numerator and denominator polynomials. 
We would then have 


Bq) _ big bag OO ee gg 


G(q, 9) = = : (57.70) 
q F(q) L+ fig t---+fuyp 
Then 
y(t) = G(q, 9)u(t) (57.71) 
is a shorthand notation for the relationship 
y(t) +fin(t — Ll) +++ +fupn(t — nf) = bi u(t — nk) +++» + Byy(t — (nb + nk — 1)) (57.72) 


There is also a time delay of nk samples. We assume assume, for simplicity, that the sampling interval T 
is one time unit. 
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In the same way the disturbance transfer function can be written 


HGS P= bebe neh Gie (57.73) 
> D(q) 1t+diq7!+---+dnaq-™ E 


The parameter vector 9 thus contains the coefficients bj, cj, dj, and f; of the transfer functions. This 
ready-made model is thus described by five structural parameters: nb, nc, nd, nf, and nk. When these 
have been chosen, it remains to adjust the parameters bj, c;, dj, and fj to data. This is done with the methods 
of Section 57.2. The ready-made model Equations 57.70 through 57.73 gives 


Bq) C(q) 
y(t) = Fg? + D@” (57.74) 
and is known as the Box-Jenkins (BJ) model, after the statisticians G. E. P. Box and G. M. Jenkins. 

An important special case is when the properties of the disturbance signals are not modeled, and the 
noise model H(q) is chosen to be H(q) = 1; that is, nc = nd = 0. This special case is known as an output 
error (OE) model since the noise source e(t) will then be the difference (error) between the actual output 
and the noise-free output: 


y(t)= mei + e(t) (57.75) 


A common variant is to use the same denominator for G and H: 
F(q) = D(q) = A(q) =14+a1q°! +++ +4naa~™ (57.76) 
Multiplying both sides of Equation 57.74 by A(q) then gives 
A(q)y(t) = B(q)u(t) + C(qe(t) (57.77) 


This ready-made model is known as the ARMAX model. The name is derived from the fact that A(q) y(t) 
represents an AutoRegression and C(q)e(t) a Moving Average of white noise, while B(q)u(t) represents 
an eXtra input (or with econometric terminology, an eXogenous variable). 

Physically, the difference between ARMAX and BJ models is that the noise and input are subjected to the 
same dynamics (same poles) in the ARMAX case. This is reasonable if the dominating disturbances enter 
early in the process (together with the input). Consider for example an airplane where the disturbances 
from wind gusts give rise to the same type of forces on the airplane as the deflections of the control 
surfaces. 

Finally, we have the special case of Equation 57.77 that C(q) = 1, that is, nc = 0 


A(q)y(t) = B(q)u(t) + e(t) (57.78) 


which with the same terminology is called an ARX model, and which we discussed at length in Sec- 
tion 57.1.3. Figure 57.1 shows the most common model structures. 

To use these ready-made models, decide on the orders na, nb, nc, nd, nf, and nk and let the computer 
pick the best model in the class thus defined. The obtained model is then scrutinized, and it might be 
found that other order must also be tested. 

A relevant question is how to use the freedom that the different model structures give. Each of the BJ, 
OE, ARMAX, and ARX structures offer their own advantages, and we will discuss them in Section 57.8.2. 


57.3.2.2 Prediction 


Starting with model (Equation 57.74), it is possible to predict what the output y(t) will be, based on 
measurements of u(s), y(s) s < t — 1, using the general formula (Equation 57.64). It is easiest to calculate 
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FIGURE 57.1 Model structures. 


the prediction for the OE-case, H(q, 9) = 1, when we obtain the model 
y(t) = G(q, 9)u(t) + e(t) 
with the natural prediction (1 — H -l=0) 
(E19) = G(q, O)u(t) (57.79) 


From the ARX case (Equation 57.78) we obtain 


y(t) = —ayy(t — 1) —- ++ — aygy(t — na) + by u(t — nk) +---+b,pu(t —nk—nb+1)+e(t) (57.80) 


and the prediction (delete e(t)!) 


y(t |9) = —ayy(t — 1) —- ++ — dnay(t — na) + by u(t — nk) +---+ bypu(t — nk — nb+ 1) (57.81) 


Note the difference between Equations 57.79 and 57.81. In the OE model the prediction is based entirely 
on the input {u(t)}, whereas the ARX model also uses old values of the output. 
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57.3.2.3 Linear Regression 


Both tailor-made and ready-made models describe how the predicted value of y(t) depends on old values 
of y and u and on the parameters 9. We denote this prediction by 


p(t|9) 


See Equation 57.64. In general this can be a rather complicated function of 8. The estimation work is 
considerably easier if the prediction is a linear function of 6: 


j(t|0) = 07 o(t) (57.82) 


Here 0 is a column vector that contains the unknown parameters, while ¢(t) is a column vector formed 
by old inputs and outputs. Such a model structure is called a linear regression. We discussed such models 
in Section 57.1.3, and noted that the ARX model (Equation 57.78) is one common model of the linear 
regression type. Linear regression models can also be obtained in several other ways. See Example 57.2. 


57.4 Frequency Domain Techniques for Linear Models 


It gives a useful complementary view as well as algorithms to consider estimating a linear model (Equa- 
tion 57.61) using frequency domain data. Such techniques are carefully and comprehensively described 
in Schoukens and Pintelon (1991) and Pintelon and Schoukens (2001). 


57.4.1 Frequency Domain Data 
The data could either be Fourier transforms of inputs and outputs or estimates of the frequency function: 
Frequency domain data from sampled measurements: 

ZN = (Uy le) Yue esis Une), Yup ee) (57.83) 


where Uy and Yy are the discrete time Fourier transforms of sampled inputs and outputs, with sampling 
interval T. 


N 
? 1 . 
Un (ei?) = — »: ulkDe Ort (57.84) 
N k=1 


If these transforms are computed on the “DFT-grid” 
= 2nk/(NT), k=0,1,...,N—1 (57.85) 


the data (Equation 57.83) will become the DFT (discrete Fourier transform) of the time domain data 
(with a special normalization). 

It is interesting to note (see, e.g., Theorem 14.25 in Pintelon and Schoukens (2001)) that under weak 
assumptions Uy(w) and Yy(w) will have an asymptotically (as N — oo) normal distribution. The values 
will also be (asymptotically) independent at different values of w; on the DFT grid. 


Measurements of the sampled frequency response function: 


We assume that we have measurements of a sampled-data frequency function G(e!”"): 
ZN = {Gmn(el 7), ..., Gn(e*)} (57.86a) 
ZY = {W(io1),...» W(ion)} (57.86b) 


where the values W are some kind of reliability measure of measurements. The frequency function 
estimates G,, in Equation 57.86 can be directly measured by certain hardware equipment, frequency 
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analyzers. Such an equipment could implement Fourier analysis as in Equation 57.87 below, or could 
rely upon the definition of frequency responses by directly measuring phase and amplitude shifts for a 
number of different sinusoidal inputs. 

The frequency responses can also be estimated/constructed from measured data either in the time or the 
frequency domain. This is the topic of Spectral Analysis, which is further dealt with in Section 57.5.2. Let 
us comment on the simplest case of spectral analysis, the Empirical Transfer Function Estimate, (ETFE). 
It is formed as the ratio of the output and input Fourier transforms 

5 (glut _ Ywlel*") 

Gy(el@T) = REDS (57.87) 
in the discrete time case. If the observations y and u have been obtained from a noise-corrupted linear 
system with frequency function Go(iw) it can be shown that the ETFE has the following statistical 
properties: Lemma 6.1 in Ljung (1999). 


ee: ; P1 
EGn (iw) = Go(io) + ———— (57.88a) 
oe" “NU (ie) 
A : ®,(w) 02 
E|Gy (io) — Goli = t 57.88b 

NEO) GEO = TE Gaye * NOW Gare Chae) 

Here ®,(«) is the spectrum of the additive noise (at the output of the system) and p; are constant bounds 

that depend on the impulse response of the system, the bound on the input, and the covariance function 
of the noise. 


All this means that we can think of the ETFE as a “noisy measurement” of the frequency function: 


Gy (ig) = Goliwx) + VE (57.89) 
with v; being a zero mean random variable with variance ®,(w,)/|Un(@ ) |?. We have then ignored the 
terms with p in the expressions above. Note that the variance of v;, would correspond to the uncertainty 
estimate W(iw,;) in Equation 57.86b. 

Something must also be said about the frequency grid in Equation 57.89: If the Fourier transforms are 
obtained by DFT of equidistantly sampled data, the natural frequencies to use in Equation 57.89 are the 
DFT grid: 


We =2kn/(NT); k=0,...,.N—-1 (57.90) 
This gives two advantages: 


¢ Frequencies in between these grid points carry no extra information: they are merely (trigonomet- 
ric) interpolations of the values on the DFT grid. This also determines the maximum frequency 
resolution of the frequency function. 

* vz are (asymptotically) uncorrelated on this grid. 


In the case of p outputs, vy is a column vector and ®, is a p x p matrix. 
The expression (Equation 57.87) assumes that u is a scalar (single input system). Formulas for multi- 
input systems are given in Ljung (2003b). 


57.4.2 Fitting Models to Fourier Transformed Data 


Consider now the case that we are given Fourier transform values of the inputs and the outputs as in 
Equation 57.83. The relationship between these values is obtained from Equation 57.55 (with Go denoting 
the “true value”): 


Yn(e®") pa Go(ei?")Un(e!®") os Vu (el?) 


B|V, ioT a 
N(e | = ©,(@) (57.91) 
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where V is the transform of the noise, corresponding to Equation 57.84. The relationship is not exact, only 
approximate, since there are transients and deviations due to the fact that the data may not be periodic. 
The relation (Equation 57.91) is like a measurement equation with uncertainties 


y(t) = Ob(t) + e(t), Var(e(t)) = ay (57.92) 


and applying least squares, with (optimal) weights being inverse error variances gives the estimation 
method 
min) 7 y(t) — 6(E)?/ou (57.93) 


In our case we obtain a criterion 


N 
Vay(0,2) = Yl Yao") — G(el*?, ey ol") i (04) (57.94) 
k=1 


In case the spectrum for v is not given, but parameterized as in Equation 57.58, we get a parameterized 
weighting, which should be balanced that as in: 


N 
Viv(0, 28) = D> YK — Gel", O) UK? / LH (e!*", 01?) 
k=1 
N 
+) log X|H(e'*", 0)|? (57.95) 
k=1 


where, for short, x = Y(e!k!). Indeed, this will be the true log-likelihood function in case ye are 
Gaussian distributed and independent for different k. 


Remark 


It may be noted that 
Tv 
i; log |H(e')| =0 (57.96) 
—T 
for any monic, stable, and inversely stable transfer function H. This means that the last sum in Equation 
57.95 is almost 8-independent for large N and equidistant frequency points. 


57.4.3 Fitting to Frequency Response Data 


Suppose now that the data is given in terms of measured frequency-response function values, Equation 
57.86. A clear-cut curve-fitting approach to estimating the model would be to form the analog of Equation 
57.24: 


N 
V0, 2%) = S° |Gmel® — (ei, 0)? / W (ierx) (57.97) 
k=1 
where we used the uncertainty measure in Equation 57.86b for the weights. 
In the case that the frequency function estimate is an ETFE as in Equation 57.87 we would use the 
uncertainty measure Equation 57.88b. This gives 
Yn(i * |Un(iw)|? 
vio) ~ Glia, 6) |Un io) | 
Un (ia) B,(w) 
= |Yy (iw) — Gliw) Un (io)? /y(w) 


IGm(im) — Gio, 8)|?/W (iw) = 


This means that for these frequency function estimates, the criterion Equation 57.97 exactly coincides 
with Equation 57.65. 
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57.4.4 Connections between Time and Frequency Domains 


Let us show the relationship between frequency domain fit (Equation 57.94) and the time domain fit 
(Equation 57.65) using Parseval’s relationship. The Fourier transform (Equation 57.84) of the prediction 
error is (neglecting transients or assuming periodic data): 


En(e®?, 0) =H !(elT 0)(Yn (el?) = G(el®T 0) Un (e!®")) 


Applying Parseval’s relationship to Equation 57.65 and ignoring transient effects (or assuming periodic 
data) now gives for this criterion 


N 
VO,ZN) = D0 1¥n Cel") — G(elT, 8) Uy (el) [7 /|H(e*™, 0)? (57.98) 
k=1 


Dividing this expression by and using Equation 57.58 we see that this expression is exactly equal to 
Equation 57.94. Consequently, also the time domain expression (Equation 57.65) can be interpreted as 
curve fitting the parameterized model to the ETFE. We have also displayed the nature of the noise model 
in Equation 57.55: it just provides the weighting in this fit. See McKelvey (2000) and the special session 
Schoukens et al. (2004), Ljung (2004) for a closer discussion on time and frequency domain connections. 


57.5 Special Estimation Techniques for Linear Black-Box Models 


An important feature of a linear, time invariant system is that it is entirely characterized by its impulse 
response. So if we know the system’s response to an impulse, we will also know its response to any 
input. Equivalently, we could study the frequency response, which is the Fourier transform of the impulse 
response. 

In this section, we shall consider estimation methods for linear systems, that do not use particular 
model parameterizations. First, in Section 57.5.1, we shall consider direct methods to determine the 
impulse response and the frequency response, by simply applying the definitions of these concepts. 

In Section 57.5.2, spectral analysis for frequency function estimation will be discussed. Finally, in 
Section 57.5.3, a recent method to estimate general linear systems (of given order, by unspecified structure) 
will be described. 


57.5.1 Transient and Frequency Analysis 


57.5.1.1 Transient Analysis 


The first step in modeling is to decide which quantities and variables are important to describe what 
happens in the system. A simple and common kind of experiment that shows how and in what time 
span various variables affect each other is called step-response analysis or transient analysis. In such 
experiments the inputs are varied (typically one at a time) as a step: u(t) = uo, t < to; u(t) = m1, t > to. 
The other measurable variables in the system are recorded during this time. We thus study the step 
response of the system. An alternative would be to study the impulse response of the system by letting the 
input be a pulse of short duration. From such measurements, information of the following nature can be 
found: 


1. The variables affected by the input in question. This makes it easier to draw block diagrams for the 
system and to decide which influences can be neglected. 

2. The time constants of the system. This also allows us to decide which relationships in the model 
can be described as static (i.e., they have significantly faster time constants than the time scale we 
are working with). 
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3. The characteristic (oscillatory, poorly damped, monotone, and the like) of the step responses, as 
well as the levels of static gains. Such information is useful when studying the behavior of the 
final model in simulation. Good agreement with the measured step responses should give a certain 
confidence in the model. 

57.5.1.2 Frequency Analysis 


If a linear system has the transfer function G(q) and the input is 
u(t) =ugcoswkT, (k—1)T <t<kT (57.99) 


then the output after possible transients have faded away will be 


y(t) =yocos(wt+), for t=T,2T,3T,... (57.100) 

where 
yo = |G(e"")| - uo (57.101) 
= arg Ge") (57.102) 


If the system is driven by the input (Equations 57.99) for a certain ug and w; and we measure yo and @ 
from the output signal, it is possible to determine the complex number G(e!") using Equation 57.101 
and Equation 57.102. By repeating this procedure for a number of different w, we can get a good estimate 
of the frequency function G(e""). This method is called frequency analysis. Sometimes it is possible to see 
or measure ug, yo, and ¢ directly from graphs of the input and output signals. Most of the time, however, 
there will be noise and irregularities that make it difficult to determine ¢ directly. A suitable procedure is 
then to correlate the output with cos wt and sin wt. 


57.5.2 Estimating the Frequency Response by Spectral Analysis 
57.5.2.1 Definitions 


(In this section, the sampling interval T is assumed to be one time unit; T = 1.) The cross spectrum 
between two (stationary) signals u(t) and y(t) is defined as the Fourier transform of their cross covariance 
function, provided this exists: 


Pyu(@) = D> Ryu(t)e'* (57.103) 
where Ry,(t) is defined by 
Ryu(t) = E(y(t) — Ey(t))(u(t — 1) — Eu(t — 1) (57.104) 


This cross covariance function is typically estimated as 


N 
“ 1 
RO = d y(thutt = 0) (57.105) 


The (auto) spectrum ®,,(w) of a signal u is defined as ®,,,,(«), that is, as its cross spectrum with itself. 
The spectrum describes the frequency contents of the signal. The connection to more explicit Fourier 
techniques is evident by the following relationship: 


,() = Jim |Un(w)|? (57.106) 


where Uy is the discrete time Fourier transform (Equation 57.84). The relationship (Equation 57.106) is 
shown in Ljung and Glad (1994). 
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Consider now the general linear model (Equation 57.55): 
y(t) = G(q)u(t) + v(t) 


It is straightforward to show that the relationships between the spectra and cross spectra of y and u 
(provided u and v are uncorrelated) is given by 


®yy(w) = Ge") &,() (57.107) 
,() = |G(e) |? ®,(@) + ®,(w) (57.108) 


It is easy to see how the transfer function G(e!®) and the noise spectrum ¢$,(w) can be estimated using 
these expressions, if only we have a method to estimate cross spectra. 


57.5.2.2 Estimation of Spectra 


The spectrum is defined as the Fourier transform of the correlation function. A natural idea would then 
be to take the transform of the estimate ah (t) in Equation 57.105. That will not work in most cases, 


though. The reason could be described as follows: the estimate RN A(t) is not reliable for large t, since 
it is based on only a few observations. These “bad” estimates are mixed with good ones in the Fourier 
transform, thus creating an overall bad estimate. It is better to introduce a weighting, so that correlation 
estimates for large lags t carry a smaller weight: 


ON,(w) = so Rt N (0) - wy (ee (57.109) 
f=-y 

This spectral estimation method is known as the The Blackman-Tukey approach. Here wy() is a window 
function that decreases with |t|. This function controls the trade-off between frequency resolution and 
variance of the estimate. A function that gives significant weights to the correlation at large lags will be 
able to provide finer frequency details (a longer time span is covered). At the same time it will have to 
use “bad” estimates, so the statistical quality (the variance) is poorer. We shall return to this trade-off in 
a moment. How should we choose the shape of the window function w,(¢)? There is no optimal solution 
to this problem, but the most common window used in spectral analysis is the Hamming window: 


1 k 
Wy(k) = 5 (1400s =) Ikl<y 


wy(k) =O0|k|>y 


(57.110) 


From the spectral estimates ,, ®y and ®y, obtained in this way, we can now use Equation 57.107, to 
obtain a natural estimate of the frequency function G(e'): 
4,N 
; 6% (w) 


G iw a 
w(e™) Na) 


(57.111) 


Furthermore, the disturbance spectrum can be estimated from Equation 57.108 as 


: : |b, Co)? 
@N (ww) = 6 (w) — (57.112) 
, 2 () 
To compute these estimates, the following steps are performed: 
Algorithm SPA (57.113) 


1. Collect data y(k), u(k)k=1,...,N. 
2. Subtract the corresponding sample means form the data. This will avoid bad estimates at very low 
frequencies. 
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3. Choose the width of the lag window w,/(k). 
Compute RN(k), RN (k), and RAK) for |k| < y according to Equation 57.105. 

5. Form the spectral estimates oy (w), oN (w), and dN (o) according to Equation 57.109 and anal- 
ogous expressions. 

6. Form Equation 57.111 and possibly also Equation 57.112. 


The user only has to choose y. A good value for systems without sharp resonances is y = 20 to 30. 
Larger values of y may be required for systems with narrow resonances. 


57.5.2.3 Quality of the Estimates 


The estimates Gy and GN are formed entirely from estimates of spectra and cross spectra. Their properties 
will therefore be inherited from the properties of the spectral estimates. For the Hamming window with 
width y, it can be shown that the frequency resolution will be about 


T . : : 
—— radians/time unit (57.114) 
yv2 
This means that details in the true frequency function that are finer than this expression will be smeared 
out in the estimate. It is also possible to show that the estimate’s variances satisfy 


. P,(w) 


i y 
Var Gy(io) © 0.7- 57.115 
ar Gy (iw) No oe) ( ) 
and 
Var ON (w) 0.7: x . &2(w) (57.116) 


[“Variance” here refers to taking expectation over the noise sequence v(t).] Note that the relative variance 
in Equation 57.115 typically increases dramatically as w tends to the Nyquist frequency. The reason is 
that |G(iw)| typically decays rapidly, while the noise-to-signal ratio B)(w)/®y(w) has a tendency to 
increase as w increases. In a Bode diagram the estimates will thus show considerable fluctuations at high 
frequencies. Moreover, the constant frequency resolution (Equation 57.114) will look thinner and thinner 
at higher frequencies in a Bode diagram due to the logarithmic frequency scale. See Ljung and Glad (1994) 
for a more detailed discussion. 


57.5.2.4 Choice of Window Size 


The choice of y is a pure trade-off between frequency resolution and variance (variability). For a spectrum 
with narrow resonance peaks it is thus necessary to choose a large value of y and accept a higher variance. 
For a more flat spectrum, smaller values of y will do well. In practice a number of different values of y are 
tried out. Often we start with a small value of y and increase it successively until an estimate is found that 
balances the trade-off between frequency resolution (true details) and variance (random fluctuations). A 
typical value for spectra without narrow resonances is y = 20-30. 


57.5.3 Subspace Estimation Techniques for State-Space Models 
A linear system can always be represented in state-space form as 


x(t + 1) = Ax(t) + Bu(t) + w(t) 


(57.117) 
y(t) = Cx(t) + Du(t) + v(t) 


with white noises w and v. Alternatively, we could just represent the input-output dynamics as in 


x(t-+ 1) = Ax(t) + Bu(t) 


(57.118) 
y(t) = Cx(t) + Du(t) + v(t) 
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where the noise at the output, v, very well could be colored. It should be noted that the input-output 
dynamics could be represented with a lower order model in Equation 57.118 than in Equation 57.117 
since describing the noise character might require some extra states. 

To estimate such a model, the matrices can be parameterized in ways that are described in Section 57.6— 
either from physical grounds or as black boxes in canonical forms. 

However, there are also other possibilities: We assume that we have no insight into the particular 
structure, and we would just estimate any matrices A, B,C, and D that give a good description of the 
input-output behavior of the system. Since there are an infinite number of such matrices that describe 
the same system (the similarity transforms), we will have to fix the coordinate basis of the state-space 
realization. 

Let us for a moment assume that not only are u and y measured, but also the sequence of state vectors x. 
This would, by the way, fix the state-space realization coordinate basis. Now, with known u, y, and x, 
the model (Equation 57.117) becomes a linear regression: the unknown parameters, all of the 
matrix entries in all the matrices, mix with measured signals in linear combinations. To see this 
clearly, let 


v(t) = ee al ae F ] 


y(t) Cc D 
vel sof 
Then, Equation 57.117 can be rewritten as 
Y(t) = @@(t) + E(t) (57.119) 


From this, all the matrix elements in © can be estimated by the simple least squares method (which in the 
case of Gaussian noise and known covariance matrix coincides with the maximum likelihood method), 
as described in Section 57.1.3 The covariance matrix for E(t) can also be estimated easily as the sample 
sum of the squared model residuals. That will give the covariance matrices as well as the cross covariance 
matrix for w and v. These matrices will, among other things, allow us to compute the Kalman filter for 
Equation 57.117. Note that all of the above holds without changes for multivariable systems, that is, when 
the output and input signals are vectors. 

The only remaining problem is where to get the state vector sequence x from? It has long been known 
(Rissanen, 1974; Akaike, 1974b), that all state vectors x(t) that can be reconstructed from input-output 
data, in fact, are linear combinations of the components of the n k-step ahead output predictors 


Rt+kt), k= {1,2,...,n} (57.120) 


where n is the model order (the dimension of x). See also Appendix 4. A in Ljung (1999). We could then 
form these predictors, and select a basis among their components: 


x(t) = LY,(t) (57.121) 
p(t + 1I\t) 

¥,(t) = (57.122) 
Ht +rI|t) 


The choice of L will determine the basis for the state-space realization, and is done in such a way that it 
is well conditioned. The predictor (t + k|t) is a linear function of u(s), y(s), 1 < s < t and can efficiently 
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be determined by linear projections directly on the input-output data. (There is one complication in that 
u(t+1),...,u(t+k) should not be predicted, even if they affect y(t + k).) 

For practical reasons the predictor is approximated so that it only depends on a fixed and finite amount 
of past data, like the s; past outputs and the s2 past inputs. This means that it takes the form 


p(t+k—1t—1)=ary(t—1)+--- +05, y(t — 51) + Brut — 1) +---+ Bs, u(t — 52) (57.123) 


This predictor can then efficiently be determined by another linear least-squares projection directly on 
the input-output data. That is, set up the model 


y(t+k—1) = 02 g(t) + y, Ut) +e(t +k-1) (57.124) 


where @ and 4 are the variables in Equation 57.123 and U accounts for the influence of u(t+k—j), 
j <k) on y(t +k — 1) which should not be modeled in the predictor. 

For large enough s, this will give a good approximation of the true predictors. 

The method thus consists of the following steps: 


Basic Subspace Algorithm (57.125) 
1. Choose sj, s2,and rand form Y(t) in Equations 57.123 and 57.122 and Y as in: 
YS|- Occ ¥,00ls (57.126) 


2. Estimate the rank n of Y and determine L in Equation 57.121 so that x(t) corresponds to a well- 
conditioned basis for it. 

3. Estimate A, B,C, D and the noise covariance matrices by applying the LS method to the linear 
regression (Equation 57.119). 


What we have described now is the subspace projection approach to estimating the matrices of the 
state-space model (Equation 57.117), including the basis for the representation and the noise covariance 
matrices. There are a number of variants of this approach. See among several references, for example, 
Van Overschee and DeMoor (1996), Larimore (1983), Verhaegen (1994) or Sections 7.3 or 10.6 in Ljung 
(1999). 

The approach gives very useful algorithms for model estimation, and is particularly well suited for 
multivariable systems. The algorithms also allow numerically very reliable implementations, and typically 
produce estimated models with good quality. If desired, the quality may be improved by using the model 
as an initial estimate for the prediction error method (Equation 57.24). 

The algorithms contain a number of choices and options, like how to choose s; and r, and also how to 
carry out step number 3. There are also several “tricks” to do step 3 so as to achieve consistent estimates 
even for finite values of s;. Accordingly, several variants of this method exist. 


57.6 Physically Parameterized Models 


So far, we have treated the parameters 9 only as vehicles to give reasonable flexibility to the transfer 
functions in the general linear model (Equation 57.61). This model can also be arrived at from other 
considerations. 

Consider a continuous time state-space model 


x(t) = A(0)x(t) + B(0)u(t) (57.127a) 
y(t) = C(0)x(t) + v(t) (57.127b) 


Here x(t) is the state vector and typically consists of physical variables (such as positions and velocities 
etc.). The state-space matrices A, B, and C are parameterized by the parameter vector 9, reflecting the 
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physical insight we have into the process. The parameters could be physical constants (resistance, heat 
transfer coefficients, aerodynamical derivatives, etc.) whose values are not known. They could also reflect 
other types of insights into the system’s properties. 


Example 57.3 An Electric Motor 


Consider an electric motor with the input u being the applied voltage and the output y being the 
angular position of the motor shaft. 

A first, but reasonable approximation of the motor’s dynamics is as a first-order system from 
voltage to angular velocity, followed by an integrator: 


G = 

(3) s(s+a) 

If we select the state variables 
y(t) 
th=|* 

a al 

we obtain the state-space form 
x= i ' }«+[e]4 
a b (57.128) 


y=[1 O0]x+v 


where v denotes disturbances and noise. In this case, we thus have 


-f 


ae =[9 = [>| (57.129) 
C=([1 | 


The parameterization reflects our insight that the system contains an integration, but is in this case 
not directly derived from detailed physical modeling. Basic physical laws would in this case have 
given us how 6 depends on physical constants, such as resistance of the wiring, amount of inertia, 
friction coefficients and magnetic field constants. 


Now, how do we fit a continuous-time model (Equation 57.127a) to sampled observed data? If the 
input u(t) has been piecewise constant over the sampling interval 
u(t) =u(kT) kT <t<(k+1)T 
then the states, inputs and outputs at the sampling instants will be represented by the discrete time model 


x((k + 1)T) = A(®)x(kT) + B(O)u(kT) 


(57.130) 
y(kT) = C(0)x(kT) + V(KT) 


where 


T 
A(6) = AT, BO) = i et B(6) dt (57.131) 
0 
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This follows from solving Equation 57.127 over one sampling period. We could also further model the 
added noise term v(kT) and represent the system in the innovations form 


X((k + 1)T) = A(®)x(kT) + B(0)u(kT) + K(0)e(KT) 


7 (57.132) 
y(kT) = C(0)x(kT) + e(kT) 


where {e(kT)} is white noise. The step from Equations 57.130 through 57.132 is really a standard Kalman 
filter step: x will be the one-step ahead predicted Kalman states. A pragmatic way to think about it is as 
follows: In Equation 57.130 the term v(kT) may not be white noise. If it is colored we may separate out that 
part of v(kT) that cannot be predicted from past values. Denote this part by e(kT): it will be the innovation. 
The other part of v(kT)—the one that can be predicted—can then be described as a combination of earlier 
innovations, e(€T)  < k. Its effect on y(kT) can then be described via the states, by changing them from 
x to x, where x contains additional states associated with getting v(kT) from e(€T), k < &. 
Now Equation 57.132 can be written in input-output form (let T = 1) 


y(t) = G(q, ®)u(t) + H(q, ®)e(t) (57.133) 
with 


G(q, 8) = C(8)(qI — A)" B®) 


: “ (57.134) 
H(q,0) =1+ C(0)(qI — A(6))'K(6) 

We are thus back at the basic linear model (Equation 57.61). The parameterization of G and H in terms 
of 6 is, however, more complicated than the ones we discussed in Section 57.3.2. 

The general estimation techniques, model properties [including the characterization (Equation 57.68)], 
algorithms, and so on, apply exactly as described in Section 57.2. 

From these examples it is also quite clear that nonlinear models with unknown parameters can be 
approached in the same way. We would then typically arrive at a structure 


x(t) = f (x(t), u(t), 9) 


(57.135) 
y(t) = h(x(t), u(t), 9) + v(t). 
In this model, all noise effects are collected as additive output disturbances v(t) which is a restriction, but 
also a very helpful simplification. If we define j(t|9) as the simulated output response to Equation 57.135, 
for a given input, ignoring the noise v(t), everything that was said in Section 57.2 about parameter 
estimation, model properties, and so on, is still applicable. 


57.7 Nonlinear Black-Box Models 


In this section, we describe the basic ideas behind model structures that have the capability to cover any 
nonlinear mapping from past data to the predicted value of y(t). Recall that we defined a general model 
structure as a parameterized mapping in Equation 57.19: 


7(t|0) = g(0, Z'") (57.136) 
We consequently allow quite general nonlinear mappings g. This section will deal with some general 


principles for how to construct such mappings, and will cover Artificial Neural Networks (ANN) as a 
special case. See Sjoberg et al. (1995) and Juditsky et al. (1995) for comprehensive surveys. 
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57.7.1 Nonlinear Black-Box Structures 


Now, the model structure family (Equation 57.136) is really too general, and it turns out to be useful to 
write g as a concatenation of two mappings: one that takes the increasing number of past observations 
Z'"! and maps them into a finite dimensional vector ¢(t) of fixed dimension and one that takes this 
vector to the space of the outputs: 


j(t|8) = g(0, Z'~!) = g(g(t), 9) (57.137) 


where 
o(t) = 9(Z'') (57.138) 


Let the dimension of ¢ be d. As before, we shall call this vector the regression vector and its components 
will be referred to as the regressors. We also allow the more general case that the formation of the regressors 
is itself parameterized: 


o(t) = 9(Z'1,n) (57.139) 


which we, for short, write g(t, n). For simplicity, the extra argument y will, however, be used explicitly 
only when essential for the discussion. 

The choice of the nonlinear mapping in Equation 57.136 has thus been reduced to two partial problems 
for dynamical systems: 


1. How to choose the nonlinear mapping g(¢) from the regressor space to the output space (i.e., from 
R4 to RP). 
2. How to choose the regressors p(t) from past inputs and outputs. 


The second problem is the same for all dynamical systems, and it turns out that the most useful 
choices of regression vectors are to let them contain past inputs and outputs, and possibly also past 
predicted/simulated outputs. The regression vector will thus be of the character in (Equation 57.4). We 
now turn to the first problem. 


57.7.2 Nonlinear Mappings: Possibilities 
57.7.2.1 Function Expansions and Basis Functions 


The nonlinear mapping 
g(@, 8) (57.140) 


goes from R@ to RP for any given 8. At this point it does not matter how the regression vector ¢ is 
constructed. It is just a vector that lives in R@. 
It is natural to think of the parameterized function family as function expansions: 


g(,8) = >> O(k)ge(g) (57.141) 


where gy are the basis functions and the coefficients 0(k) are the “coordinates” of g in the chosen basis. 

Now, the only remaining question is: how to choose the basis functions g,? Depending on the support 
of gy [i.e., the area in R4 for which g;(¢) is (practically) nonzero] we shall distinguish between three types 
of basis functions 


e Global basis functions 
¢ Semiglobal or ridge-type basis functions 
¢ Local basis functions 
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57.7.2.2 Global Basis Function 


A typical and classical global basis function expansion would then be the Taylor series, or polynomial 
expansion, where g, would contain multinomials in the components of ¢ of total degree k. Fourier series 
are also relevant examples. We shall, however, not discuss global basis functions here any further. Expe- 
rience has indicated that they are inferior to the semilocal and local ones in typical practical applications. 


57.7.2.3 Local Basis Functions 


Local basis functions have their support only in some neighborhood of a given point. Think (in the case 
of p = 1) of the indicator function for the unit cube: 


K(~) = 1 if |g,%| <1 Vk, and _ 0 otherwise (57.142) 
By scaling the cube and placing it at different locations we obtain the functions 


SP) = K(x * (Y — Bx)) (57.143) 


By allowing a to be a vector of the same dimension as ¢ and interpreting the multiplication « as 
component-wise multiplication (like “.*” in MATLAB) we may also reshape the cube to be any par- 
allelepiped. The parameters a are thus scaling or dilation parameters while B determine location or 


translation. For notational convenience we write 


BK(P) = K(ay * (M — Bx)) = K(PK- G) (57.144) 


where 
Ok = [OK OK * Bx] 


In the last equality, with some abuse of notation, we expanded the regression vector ~ to contain some 
“1":s. This is to stress the point that the argument of the basic function « is bilinear in the scale and 
location parameters p;, and in the regression vector ¢~. The notation p,; - ¢ indicates this. 

This choice of gy in Equation 57.141 gives functions that are piecewise constant over areas in R4 that 
can be chosen arbitrarily small by proper choice of the scaling parameters. It should be fairly obvious that 
such functions g, can approximate any reasonable function arbitrarily well. 

Now it is also reasonable that the same will be true for any other localized function, such as the Gaussian 
bell function: 

K(g) =e !# (57.145) 


57.7.2.4 Ridge-Type Basis Functions 


A useful alternative is to let the basis functions be local in one direction of the g-space and global in the 
others. This is achieved quite analogously to Equation 57.143 as follows. Let o(x) be a local function from 
Rto R. Then form 


g() = o(4g (P — Bx)) = O(a, P+ YE) = 0( OK - ©) (57.146) 


where the scalar yz, = —af Bes and 
Ok = [8K Ve] 


Note the difference with Equation 57.143! The scalar product at @ is constant in the subspace of R@ that 
is perpendicular to the scaling vector ax. Hence, the function g;(¢) varies like o in a direction parallel to 
a, and is constant across this direction. This motivates the term semiglobal or ridge-type for this choice 
of functions. 

As in Equation 57.143 we expanded in the last equality in Equation 57.146 the vector ~ with the value 
“I,” again just to emphasize that the argument of the fundamental basis function o is bilinear in p and 9. 


57-32 Control System Advanced Methods 


57.7.2.5 Connection to “Named Structures” 


Here we briefly review some popular structures, other structures related to interpolation techniques are 
discussed in Sjéberg et al. (1995), Juditsky et al. (1995). 


57.7.2.5.1 Wavelets 


The local approach corresponding to Equations 57.141 and 57.143 has direct connections to wavelet 
networks and wavelet transforms. The exact relationships are discussed in Sjoberg et al. (1995). Loosely, 
we note that via the dilation parameters in px we can work with different scales simultaneously to pick 
up both local and not-so-local variations. With appropriate translations and dilations of a single suitably 
chosen function k (the “mother wavelet”), we can make the expansion (Equation 57.141) orthonormal. 
This is discussed extensively in Juditsky et al. (1995). 


57.7.2.5.2. Wavelet and Radial Basis Networks 

The choice (Equation 57.145) without any orthogonalization is found in both wavelet networks (Zhang 
and Benveniste, 1992) and radial basis neural networks (Poggio and Girosi, 1990). 

57.7.2.5.3, Neural Networks 

The ridge choice (Equation 57.146) with 


aie l+e* 


gives a much used neural network structure, namely, the one hidden layer feedforward sigmoidal net. 


57.7.2.5.4 Hinging Hyperplanes 


If instead of using the sigmoid o function we choose “V-shaped” functions (in the form of a higher- 
dimensional “open book”) Breiman’s hinging hyperplane structure is obtained, (Breiman 1993). Hinging 
hyperplanes model structures (Breiman 1993) have the form 


g(x) = max {Btx+y7,B x+y } or g(x) =min {Btx+y",B x+y I. 
It can be written in a different way: 


1 


gtx) = S18 +B xt yt ty T+ 516 -B xt yt 7". 


Thus a hinge is the superposition of a linear map and a semiglobal function. Therefore, we consider hinge 
functions as semiglobal or ridge-type, though it is not in strict accordance with our definition. 


57.7.2.5.5 Nearest Neighbors or Interpolation 


By selecting k as in Equation 57.142 and the location and scale vector px in the structure (Equation 57.143), 
such that exactly one observation falls into each “cube,” the nearest-neighbor model is obtained: just load 
the input-output record into a table, and, for a given @, pick the pair (y, @) for @ closest to the given 9, ¥ 
is the desired output estimate. If one replaces Equation 57.142 by a smoother function and allows some 
overlapping of the basis functions, we get interpolation type techniques such as kernel estimators. 


57.7.2.5.6 Fuzzy Models 


Also so called fuzzy models based on fuzzy set membership belong to the model structures of the class 
(Equation 57.141). The basis functions g, then are constructed from the fuzzy set membership functions 
and the inference rules. The exact relationship is described in Sjoberg et al. (1995). 
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57.7.3 Estimating Nonlinear Black-Box Models 
The model structure is determined by the following choices: 


¢ The regression vector (typically built up from past inputs and outputs) 
¢ The basic function k (local) or o (ridge) 
¢ The number of elements (nodes) in the expansion Equation 57.141 


Once these choices have been made (t|9) = g((t), 9) is a well-defined function of past data and the 
parameters 0. The parameters are made up of coordinates in the expansion Equation 57.141, and from 
location and scale parameters in the different basis functions. 

All the algorithms and analytical results of Section 57.2 can thus be applied. For Neural Network appli- 
cations these are also the typical estimation algorithms used, often complemented with regularization, 
which means that a term is added to the criterion (Equation 57.24), that penalizes the norm of 0. This 
will reduce the variance of the model, in that “spurious” parameters are not allowed to take on large, and 
mostly random values. See, for example, (Sjoberg et al. 1995). 

For wavelet applications it is common to distinguish between those parameters that enter linearly in 
¥(t|9) (i-e., the coordinates in the function expansion) and those that enter nonlinearly (ie., the location 
and scale parameters). Often the latter are seeded to fixed values and the coordinates are estimated by 
the linear least squares method. Basis functions that give a small contribution to the fit (corresponding 
to nonuseful values of the scale and location parameters) can then be trimmed away (“pruning” or 
“shrinking”). 


57.8 User’s Issues 


57.8.1 Experiment Design 


It is desirable to affect the conditions under which the data are collected. The objective with such experi- 
ment design is to make the collected data set Z' as informative as possible with respect to the models to 
be built using the data. A considerable amount of theory around this topic can be developed and we shall 
here just review some basic points. An inspiring discussion on experiment design is given in Hjalmarsson 
(2005). 

The first and most important point is the following one: 


1. The input signal u must be such that it exposes all the relevant properties of the system. It must thus 
not be too “simple.” For example, a pure sinusoid 


u(t) = Acos wt 


will only give information about the system’s frequency response at frequency w. This can also be 
seen from Equation 57.68. The rule is that 
¢ The input must contain at least as many different frequencies as the order of the linear model 
to be built. 
« To be on the safe side, a good choice is to let the input be random (such as filtered white 
noise). It then contains all frequencies. 
Another case where the input is too simple is when it is generated by feedback such as 


u(t) = —Ky(t) (57.147) 
If we would like to build a first-order ARX model 


y(t) + ay(t — 1) = bu(t — 1) + e(t) 
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we find that for any given a all models such that 
a+bK=a 


will give identical input-output data. We can thus not distinguish between these models using an 
experiment with Equation 57.147. That is, we cannot distinguish between any combinations of “a” 
and “b” if they satisfy the above condition for a given “a.” The rule is 

« Ifclosed-loop experiments have to be performed, the feedback law must not be too simple. 


It is to be preferred that a set-point in the regulator is being changed in a random fashion. 
The second main point in experimental design is 


2. Allocate the input power to those frequency bands where a good model is particularly important. 
This is also seen from the expression (Equation 57.68). 


If we let the input be filtered white noise, this gives information how to choose the filter. In the time 
domain it is often useful to think like this: 


¢ Use binary (two-level) inputs if linear models are to be built: this gives maximal variance for 
amplitude-constrained inputs. 

¢ Check that the changes between the levels are such that the input occasionally stays on one level so 
long that a step response from the system has time, more or less, to settle. There is no need to let 
the input signal switch so quickly back and forth that no response in the output is clearly visible. 


Note that the second point is really just a reformulation in the time domain of the basic frequency domain 
advice: let the input energy be concentrated in the important frequency-bands. 
A third basic piece of advice about experiment design concerns the choice of sampling interval. 


3. Atypical good sampling frequency is 10 times the bandwidth of the system. That corresponds roughly 
to 5-7 samples along the rise time of a step response. 


57.8.2 Model Validation and Model Selection 


The system identification process has, as we have seen, these basic ingredients 


e The set of models 
e The data 
e The selection criterion 


Once these have been decided upon, we have, at least implicitly, defined a model: the one in the set that 
best describes the data according to the criterion. It is thus, in a sense, the best available model in the 
chosen set. But is it good enough? It is the objective of model validation to answer that question. Often the 
answer turns out to be “no,” and we then have to go back and review the choice of model set, or perhaps 
modify the data set. See Figure 57.2. 

How do we check the quality of a model? The prime method is to investigate how well it is capable 
of reproducing the behavior of a new set of data (the validation data) that was not used to fit the model. 
That is, we simulate the obtained model with a new input and compare this simulated output. One may 
then use one’s eyes or numerical measurements of fit to decide if the fit in question is good enough. 
Suppose we have obtained several different models in different model structures (say a fourth order ARX 
model, a second order BJ model, a physically parameterized one, etc.) and would like to know which 
one is best. The simplest and most pragmatic approach to this problem is then to simulate each one of 
them on validation data, evaluate their performance, and pick the one that shows the most favorable fit 
to measured data. (This could indeed be a subjective criterion!) 
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FIGURE 57.2 The identification loop. 


The second basic method for model validation is to examine the residuals (“the leftovers”) from the 
identification process. These are the prediction errors 


e(t) = e(t, Oy) = y(t) — 3(t|6n) 


that is, what the model could not “explain.” Ideally these should be independent of information that was 
at hand at time ft — 1. For example, if e(t) and u(t — t) turn out to be correlated, then there are things 
in y(t) that originate from u(t — t) but have not been properly accounted for by (t|6n). The model has 
then not squeezed out all relevant information about the system from the data. 

It is good practice to always check the residuals for such (and other) dependencies. This is known as 
residual analysis. 


57.8.3 Software for System Identification 


In practice System Identification is characterized by some quite heavy numerical calculations to determine 
the best model in each given class of models. This is mixed with several user choices, trying different 
model structures, filtering data, and so on. In practical applications we will thus need good software 
support. There are now many different packages for identification available, such as Mathwork’s System 
Identification Toolbox (Ljung, 2007), Matrixi,s System Identification Module (MATRIX,, 1991), PIM 
(Landau, 1990), UNIT (Ninness and Wills, 2006) and CONTSID (Garnier and Mensler, 2000). They all 
have in common that they offer the following routines: 


A Handling of data, plotting, and so on. 

Filtering of data, removal of drift, choice of data segments, and so on. 
B Nonparametric identification methods 

Estimation of covariances, Fourier transforms, correlation-, and spectral-analysis, and so on. 
C Parametric estimation methods 

Calculation of parametric estimates in different model structures. 
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D_ Presentation of models 
Simulation of models, estimation and plotting of poles and zeros, computation of frequency 
functions, and plotting Bode diagrams, and so on. 

E Model validation 
Computation and analysis of residuals (e(t, 6n)). Comparison between different models’ prop- 
erties, and so on. 


The existing program packages differ mainly in various user interfaces and by different options regard- 
ing the choice of model structure according to C above. For example, MATLAB’s Identification Toolbox 
(Ljung, 2007) covers all linear model structures discussed here, including arbitrarily parameterized linear 
models in continuous time. 

Regarding the user interface, there is now a clear trend to make it graphically oriented. This avoids 
syntax problems and relies more on “click and move,” at the same time as tedious menu-labyrinths are 
avoided. More aspects of CAD tools for system identification are treated in Ljung (2003a). 


57.8.4 The Practical Side of System Identification 


It follows from our discussion that the most essential element in the process of identification—once the 
data have been recorded—is to try out various model structures, compute the best model in the structures, 
using Equation 57.24, and then validate this model. Typically this has to be repeated with quite a few 
different structures before a satisfactory model can be found. 

The difficulties of this process should not be underestimated, and it will require substantial experience 
to master it. Here follows, however, a procedure that could prove useful to try out. 


Step 1: Looking at the data. Plot the data. Look at them carefully. Try to see the dynamics with your 

own eyes. Can you see the effects in the outputs of the changes in the input? Can you see nonlinear 
effects, like different responses at different levels, or different responses to a step up and a step 
down? Are there portions of the data that appear to be “messy” or carry no information. Use this 
insight to select portions of the data for estimation and validation purposes. 
Do physical levels play a role in your model? If not, detrend the data by removing their mean 
values. The models will then describe how changes in the input give changes in output, but not 
explain the actual levels of the signals. This is the normal situation. The default situation, with good 
data, is that you detrend by removing means, and then select the first two-thirds or so of the data 
record for estimation purposes, and use the remaining data for validation. (All of this corresponds 
to the “Data Quickstart” in the MATLAB Identification Toolbox.) 

Step 2: Getting a feel for the difficulties. Compute and display the spectral analysis frequency response 
estimate, the correlation analysis impulse response estimate as well as a fourth-order ARX model 
with a delay estimated from the correlation analysis and a default order state-space model com- 
puted by a subspace method. (All of this corresponds to the “Estimate Quickstart” in the MATLAB 
Identification Toolbox.) This gives three plots. Look at the agreement between the 

« Spectral analysis estimate and the ARX and state-space models’ frequency functions. 
¢ Correlation analysis estimate and the ARX and state-space models’ transient responses. 
« Measured Validation Data output and the ARX and state-space models’ simulated outputs. 
We call this the Model Output Plot. 
If these agreements are reasonable, the problem is not so difficult, and a relatively simple linear 
model will do a good job. Some fine tuning of model orders, and noise models have to be made 
and you can proceed to Step 4. Otherwise go to Step 3. 

Step 3: Examining the difficulties. There may be several reasons why the comparisons in Step 2 did not 
look good. This section discusses the most common ones, and how they can be handled: 

« Model unstable: The ARX or state-space model may turn out to be unstable, but could still 
be useful for control purposes. Then change to a 5- or 10-step ahead prediction instead of 
simulation in the Model Output Plot. 
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¢ Feedback in data: If there is feedback from the output to the input, due to some regulator, 
then the spectral and correlations analysis estimates are not reliable. Discrepancies between 
these estimates and the ARX and state-space models can therefore be disregarded in this 
case. In residual analysis of the parametric models, feedback in data can also be visible as 
correlation between residuals and input for negative lags. 

« Noise model: If the state-space model is clearly better than the ARX model at reproducing 
the measured output this is an indication that the disturbances have a substantial influence, 
and it will be necessary to carefully model them. 

¢ Model order: If a fourth-order model does not give a good Model Output plot, try eighth 
order. If the fit clearly improves, it follows that higher-order models will be required, but 
that linear models could be sufficient. 

« Additional inputs: If the Model Output fit has not significantly improved by the tests so far, 
think over the physics of the application. Are there more signals that have been, or could 
be, measured that might influence the output? If so, include these among the inputs and try 
again a fourth-order ARX model from all the inputs. (Note that the inputs need not at all be 
control signals, anything measurable, including disturbances, should be treated as inputs). 

« Nonlinear effects: If the fit between measured and model output is still bad, consider the 
physics of the application. Are there nonlinear effects in the system? In that case, form the 
nonlinearities from the measured data. This could be as simple as forming the product of 
voltage and current measurements, if you realize that it is the electrical power that is the 
driving stimulus in, say, a heating process, and temperature is the output. This is of course 
application dependent. It does not cost very much work, however, to form a number of 
additional inputs by reasonable nonlinear transformations of the measured ones, and just 
test if inclusion of them improves the fit. See Example 57.2. 

¢ Still problems? If none of these tests leads to a model that is able to reproduce the Validation 
Data reasonably well, the conclusion might be that a sufficiently good model cannot be 
produced from the data. There may be many reasons for this. The most important one is 
that the data simply do not contain sufficient information, for example, due to bad signal 
to noise ratios, large and nonstationary disturbances, varying system properties, and so on. 
The reason may also be that the system has some quite complicated nonlinearities, which 
cannot be realized on physical grounds. In such cases, nonlinear, black-box models could be 
a solution. Among the most used models of this character are the ANN. See Section 57.7. 

Otherwise, use the insights on which inputs to use and which model orders to expect and proceed 
to Step 4. 

Step 4: Fine tuning orders and noise structures. For real data there is no such thing as a “correct model 
structure.” However, different structures can give quite different model quality. The only way to 
find this out is to try out a number of different structures and compare the properties of the 
obtained models. There are a few things to look for in these comparisons: 

« Fit between simulated and measured output: Look at the fit between the model’s simu- 
lated output and the measured one for the Validation Data. Formally, you could pick that 
model, for which this number is the lowest. In practice, it is better to be more pragmatic, 
and also take into account the model complexity, and whether the important features of the 
output response are captured. 

¢ Residual analysis test: You should require of a good model so that the cross correlation 
function between residuals and input does not go significantly outside the confidence region. 
A clear peak at lag k shows that the effect from input u(t — k) on y(t) is not properly described. 
A rule of thumb is that a slowly varying cross correlation function outside the confidence 
region is an indication of too few poles, while sharper peaks indicate too few zeros or wrong 
delays. 
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¢ Pole zero cancellations: If the pole-zero plot (including confidence intervals) indicates 
pole-zero cancellations in the dynamics, this suggests that lower-order models can be used. 
In particular, if it turns out that the order of ARX models has to be increased to get a good 
fit, but that pole-zero cancellations are indicated, then the extra poles are just introduced to 
describe the noise. Then try ARMAX, OE, or BJ model structures with an A or F polynomial 
of an order equal to that of the number of noncanceled poles. 
What Model Structures Should be Tested? Well, you can spend any amount of time to check 
out a very large number of structures. It often takes just a few seconds to compute and evaluate a 
model in a certain structure, so that you should have a generous attitude to the testing. However, 
experience shows that when the basic properties of the system’s behavior have been picked up, it 
is not of much use to fine-tune orders in absurdum just to improve the fit by fractions of percents. 
For ARX models and state-space models estimated by subspace methods there are also efficient 
algorithms for handling many model structures in parallel. 
Multivariable Systems: Systems with many input signals and/or many output signals are called 
multivariable. Such systems are often more challenging to model. In particular systems with several 
outputs could be difficult. A basic reason for the difficulties is that the couplings between several 
inputs and outputs lead to more complex models: The structures involved are richer and more 
parameters will be required to obtain a good fit. 

Generally speaking, it is preferable to work with state-space models in the multivariable case, 

since the model structure complexity is easier to deal with. It is essentially just a matter of choosing 
the model order. 
Working with Subsets of the Input-Output Channels: In the process of identifying good 
models of a system it is often useful to select subsets of the input and output channels. Partial 
models of the system’s behavior will then be constructed. It might not, for example, be clear if all 
measured inputs have a significant influence on the outputs. That is most easily tested by removing 
an input channel from the data, building a model for how the output(s) depend on the remaining 
input channels, and checking if there is a significant deterioration in the model output’s fit to the 
measured one. See also the discussion under Step 3. Generally speaking, the fit gets better when 
more inputs are included and worse when more outputs are included. To understand the latter fact, 
you should realize that a model that has to explain the behavior of several outputs has a tougher job 
than one that just must account for a single output. If you have difficulties to obtain good models 
for a multioutput system, it might thus be wise to model one output at a time, to find out which 
are the difficult ones to handle. Models that just are to be used for simulations could very well be 
built up from single-output models, for one output at a time. However, models for prediction and 
control will be able to produce better results if constructed for all outputs simultaneously. This 
follows from the fact that knowing the set of all previous output channels gives a better basis for 
prediction, than just knowing the past outputs in one channel. 


Step 5: Accepting the model. The final step is to accept, at least for the time being, the model to be 


used for its intended application. Recall the answer to question 10 in the introduction: No matter 
how good an estimated model looks on your screen, has only picked up a simple reflection of reality. 
Surprisingly often, however, this is sufficient for rational decision making. 
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58.1 Caveat 


What follows is a quick survey of the main ingredients in the theory of discrete-time Markov processes. 
It is a bird’s view, rather than the definitive “state of the art.” To maximize accessibility, the nomencla- 
ture of mathematical probability is avoided, although rigor is not sacrificed. To compensate, examples 
(and counterexamples) abound and the references are annotated. Relevance to control is discussed in 
Section 58.9. 


58.2 Introduction 


Discrete time Markov processes, or Markov chains, are a powerful tool for modeling and analysis of 
discrete time systems, whose behavior is influenced by randomness. A Markov chain is probably the 
simplest object, which incorporates both dynamics (i.e., notions of “state” and time) and randomness. 
Let us illustrate the idea through a gambling example. 


Example 58.1: 


A gambler bets one dollar on “red” at every turn of a (fair) game of roulette. Then, at every turn he 
gains either one dollar (win, with probability 1/2) or (—1) (lose). In an ideal game, the gains form 
a sequence of independent, identically distributed (i.i.d.) random variables. We cannot predict the 
outcome of each bet, although we do know the odds. Denote by X; the total fortune the gambler 
has at time t. If we know Xs, then we can calculate the distribution of X;1 (i.e., the probability that 
X;41 =Y, for all possible values of y), and even of X,, for any k > 0. The variable X; serves as a 
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“state” in the following sense: given Xs, knowledge of X;¢ for t < s is irrelevant to the calculation of 
the distribution of future values of the state. 


This notion of a state is similar to classical “state space” descriptions. Consider the standard linear 
model of a dynamical system 
Xt4+1 = Axt + vt; (58.1) 


or the more general nonlinear model 
x41 =f (Xt V4). (58.2) 


It is intuitively clear that, given the present state, the past does not provide additional information about 
the future, as long as v; is not predictable (this is why deterministic state-space models require that v; be 
allowed to change arbitrarily at each t). This may remain true even when 1 is random: for example, when 
they are ii.d., for then the past does not provide additional information about future v,’s. As we shall see 
in Theorem 58.1, in this case Equations 58.1 and 58.2 define Markov chains. 

In the next section we give the basic definitions and describe the dynamics of Markov chains. We 
assume that the state space S is countable. We restrict our attention to time-homogeneous dynamics (see 
comment following Theorem 58.1), and discuss the limiting properties of the Markov chain. Finally, we 
shall discuss extensions to continuous time and to more general state spaces. We conclude this section with 
a brief review of standard notation. All of our random variables and events are defined on a probability 
space Q with a collection F of events (subsets of 2) and probability P. The “probability triple” (Q, F, P) 
is fixed, and we denote expectation by E. For events A and B with P(B) > 0, the basic definition of a 
conditional probability (the multiplication rule) is 


(58.3) 


The abbreviation i.i.d. stands for independent, identically distributed, and random variables are denoted by 
capital letters. The identity matrix is denoted by I and 1, is the indicator function of A, that is, 14() = 1 
if m € A and = 0 otherwise. 


58.3 Definitions and Construction 


Let Xo, X1,... be a sequence of random variables, with values in a state space S. We assume that S is finite 
or countable, and for convenience we usually set S = {1,2,...}. 


Definition 58.1: 


A sequence Xo, X,,...0nS is a Markov chain if it possesses the Markov property, that is, if for allt > 0 and 
all it—1> Wess » 10, 


P (Sha Hh, ee a Hh) HP HS She). 


A Markov chain is called homogeneous if P (X; = j|X:—1 = i) does not depend on t. In this case we denote 
the transition probability from i to j by 


py =P (X= JX =8). 


The (possibly infinite) matrix P 5 {py ij =1,...} is called the transition matrix. 
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The restriction to homogeneous chains is not significant: if we define a new state x S {t, x}, then it is 
not hard to see that we can incorporate explicit time dependence within a homogeneous chain, and the 
new state space is still countable. Henceforth, we restrict our attention to homogeneous chains. 

Using the Markov property, a little algebra with the definition of conditional probability gives the more 
general Markov property: if t) < t2 <---<t, <---<tg then 


P (Xty = jes Xty_, = Je-v---» Xo, = FelXyy = fev» Xn =fr) 
SP Rg Sed yp tise ap = ie ee ee) 4 (58.4) 


This is a precise statement of the intuitive idea given in the introduction: given the present state 
Xt,_; = jk—1, the past does not provide additional information. 

A chain is called finite if S is a finite set. An alternative name for “homogeneous Markov chain” 
is “Markov chain with stationary transition probabilities.” There are those who call a homogeneous 
Markov chain a “stationary Markov chain.” However, since “stationary process” means something entirely 
different, we shall avoid such usage (see Example 58.6). 

Suppose that a process is a Markov chain according to Definition 58.1, and that its initial distribution 
is given by the row vector |1(0), that is, 


P (Xp =i) = n4i(0), emt pe eee 


Then, we can calculate the joint probability distribution at times 0 and 1 from the definition of conditional 


probability, 


P (Xo =i, X1 =j) =P (Xi = j|Xo = i) - P (Xo =i) =1:(0) - py. 
More generally, using the Markov property: 
t 
P (Xo = jo. Xi =f. wes Xt = jr) = [jo (0) * | [eee 
s=1 


So, the probability distribution of the whole process can be calculated from the two quantities: the initial 
probability distribution, and the transition probabilities. In particular, we can calculate the probability 
distribution at any time t 


j(t) = P(X; =j) 


i 
= YS vp] [pias (58.5) 


JO fls-+-xft-1 s=1 
where the sum is over all states in S, that is, each index is summed over the values 1, 2,.... In vector 
notation, 
w(t) = w(t — 1)P=--- = (0)P". (58.6) 


(If S is countable then, of course, the vector and matrix P are infinite, but their product is defined 
in exactly the same way as for finite ones.) Thus, the probability distribution of a Markov chain evolves 
as a linear dynamical system, even when its evolution equation 58.2 is nonlinear. The one-dimensional 
probability distribution and the transition probabilities clearly satisfy 


ae (pee 
LjeSbjO=1  [Ljes Pi = 1. 


That is, the rows of the transition matrix P sum to one. Thus, P is a stochastic matrix: its elements are 
nonnegative and its rows sum to one. 


(58.7) 
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If we denote by py =p (Xm4n = j|Xm = i) the n step transition probability from i to j, then we obtain 
from the definition of conditional probability and the Markov property (Equation 58.4) the Chapman- 
Kolmogorov equations 


(n+m) (n)_(m) 
Py = yy Pik Px 
keS 
or equivalently 


PET cs Dipl (58.8) 
Therefore, the py are the elements of the matrix P”. This matrix notation yields a compact expression 
for expectations of functions of the state. To compute Eg(X;) for some function g on S = {1,2,...}, we 


represent g by a column vector g é {g(1),g(2),...} (where T denotes transpose). Then 


Fg(X+) = (0) Pf -g. 


Note that this expression does not depend on the particular S: since the state space is countable we can, 
by definition, relabel the states so that the state space becomes {1,2,...}. Let us now summarize the 
connection between the representations (Equations 58.1 and 58.2) and Markov chains. 


Theorem 58.1: 


Let Vo, Vi,... be a sequence of i.i.d. random variables, independent of Xo. Then for any (measurable) 
function f, the sequence Xo,X\,... defined through Equation 58.2 is a Markov chain. Conversely, let 
Xo,.X1,... be a Markov chain with values in S. Then there is a (probability triple and a measurable) 
function f and a sequence Vo, Vi,... of i.i.d. random variables so that the process Xo, X1,... defined 
by Equation 58.2 with Xy = Xo has the same probability distribution as Xo, X\,... that is, for all t and 


jorfis+++>fts 


P (Xo = jo, X1 =fr,-... Xe =r) = P (Xo =o. Xi =fr,..., Xr =jr). 


Note that, whether the system (Equation 58.2) is linear or not, the evolution of the probability distribution 
(Equations 58.5 and 58.6 ) is always linear. 

We have seen that a Markov chain defines a set of transition probabilities. The converse is also true: 
given a set of transition probabilities and an initial probability distribution, it is possible to construct a 
stochastic process which is a Markov chain with the specified transitions and probability distribution. 


Theorem 58.2: 


If Xo, X1,... is a homogeneous Markov chain then its probability distribution and transition probabilities 
satisfy Equations 58.7 and 58.8. Conversely, given \.(0) and a matrix P that satisfy Equation 58.7, there 
exists a (probability triple and a) Markov chain with initial distribution (0) and transition matrix P. 


Example 58.2: 


Let Vo, Vi,... be i.i.d. and independent of Xo. Assume both V; and Xo have integer values. Then 
re t 
X41 =X + Vt =Xo + DVs (58.9) 
s=0 
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defines a Markov chain called a chain with stationary independent increments, with state space 
..-,—1,0,1,.... The transition probability Pi depends only on the difference j — i. It turns out [1] 
that the converse is also true: if the transition probabilities of a Markov chain depend only on the 
difference j — i then the process can be obtained via Equation 58.9 with i.i.d. Vt. 


A random walk is a process defined through Equation 58.9, but where the V; are not necessarily 
integers—they are real valued. 


58.4 Properties and Classification 


Given two states i and j, it may or may not be possible to reach j from i. This leads to the notion of 
classes. 


Definition 58.2: 


We say a state i leads to j if 
P (xt = j for some t|xo = i) >0. 


This holds if and only if py > 0 for some t. We say states i and j communicate, denoted by i <> j, if i leads 
to j and j leads to i. 


Communication is a property of pairs: it is obviously symmetric (i <j if and only if j = i) and is 
transitive (i <> j andj < k implies i < k) by the Chapman-Kolmogorov equations 58.8. By convention, 
i <> i. By these three properties, <> defines an equivalence relation. We can therefore partition S into 
nonempty communicating classes So, 81, ... with the properties 


1. Every state i belongs to exactly one class 
2. Ifiandj belong to the same class then i <> j 
3. Ifiandj belong to different classes then i and j do not communicate 


We denote the class containing state i by S(i). Note that if i leads to j but j does not lead to i then i and j 
do not communicate. 


Definition 58.3: 


A set of states C is closed, or absorbing, if pj = 0 whenever i € C andj ¢ C. Equivalently, 
Se Ps =1 forallieC. 
jeCc 


If a set C is not closed, then it is called open. A Markov chain is irreducible if all states communicate. In this 
case its partition contains exactly one class. A Markov chain is indecomposable if its partition contains at 
most one closed class. 


Define the incidence matrix T as follows: 


L= 1 if pj >0 
4“ )0 otherwise. 
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We can also define a directed graph whose nodes are the states, with a directed arc between any two 
states for which Zj; = 1. The communication properties can obviously be extracted from Z or from the 
directed graph. The chain is irreducible if and only if the directed graph is connected in the sense that, 
going in the direction of the arcs, we can reach any node from any other node. Closed classes can also be 
defined in terms of the incidence matrix or the graph. The classification leads to the following maximal 
decomposition. 


Theorem 58.3: 


By reordering the states, if necessary, we can put the matrix P into the block form 


Pir O ae. 08. %0 
O:. “Pak tes. OP D 
ey eee a (58.10) 
fo "Ola Pye 0 
Bi. Res .ax)- Rw 20 


where the blocks P; correspond to closed irreducible classes. It is maximal in the sense that smaller classes 
will not be closed, and no subset of states corresponding to Q is a closed irreducible class. 


If S is countable, then the number of classes and the size of some blocks may be infinite. Note that all 
the definitions in this section apply when we replace S with any closed class. 

Much like other dynamical systems, Markov chains can have cyclic behavior, and can be unstable. The 
relevant definitions are 


Definition 58.4: 


The period d of a state i is the greatest common divisor of the set {t : pe > 0}. If d is finite and d > 1 then 
the state is called periodic; otherwise it is aperiodic. 


Definition 58.5: 


A state i is called recurrent if the probability of starting at i and returning to i in finite time is 1. Formally, if 
P (Xr =i for some t > 1|x, = i) =1. 


Otherwise it is called transient. 


Example 58.3: 


In the chain on S = {1,2} with pj = 1 if and only if i #j, both states are periodic with period d = 2 
and both states are recurrent. The states communicate, and so S contains exactly one class, which 
is therefore closed. Consequently, the chain is irreducible and indecomposable. However, if p32 = 
P22 = 1 then the states are not periodic, state 2 is recurrent and state 1 is transient. In this case, the 
partition contains two sets: the closed set {2}, and the open set {1}. Consequently, the chain is not 
irreducible, but it is indecomposable. 
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When S is finite, then either it is irreducible or it contains a closed proper subset. 


Example 58.4: 


Let S={1,2,...}. Suppose py =1 if and only if j=/+ 1. Then all states are transient, and S is 
indecomposable but not irreducible. Every set of the form {i : i => k} is closed, but in the partition of 
the state space each state is the only member in its class. Suppose now py; = 1 and fori > 1, pj =1 
if and only ifj = i— 1. Then state 1 is the only recurrent state, and again each state is alone in its class. 


Theorem 58.4: 


Let S, be a class. Then either all states in S, are recurrent, or all are transient. Moreover, all states in S, 
have the same period d. 


58.5 Algebraic View and Stationarity 


The matrix P is positive, in the sense that its entries are positive. When S is finite, the Perron—Frobenius 
theorem implies [3] 


Theorem 58.5: 


Let S be finite. Then P has a nonzero left eigenvector n whose entries are nonnegative, and n-P =n, that 
is, the corresponding eigenvalue is 1. Moreover, || < 1 for all other eigenvalues }. The multiplicity of the 
eigenvalue 1 is equal to the number of irreducible closed subsets of the chain. In particular, if the entries of 
P" are all positive for some n, then the eigenvalue 1 has multiplicity 1. In this case, the entries of x are all 
positive and || < 1 for all other eigenvalues i. 


If the entries of P” are all positive for some n then the chain is irreducible and aperiodic, hence the second 
part of the theorem. If the chain is irreducible and periodic with period d, then the d roots of unity are 
left eigenvalues of P, each is of multiplicity 1 and all other eigenvalues have strictly smaller modulus. The 
results for a general finite chain can be obtained by writing the chain in the block form (Equation 58.10). 


Definition 58.6: 


Let S be finite or countable. A probability distribution w satisfying  - P= is called invariant (under P) 
or stationary. 


Theorem 58.5 thus implies that every finite Markov chain possesses at least one invariant probability 
distribution. For countable chains, Example 58.4 shows that this is not true. 


Example 58.5: 


Returning to Example 58.3, in the first case (1/2, 1/2) is the only invariant probability distribution, 
while in the second case (0, 1) is the only invariant probability distribution. In Example 58.4, in the 
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first case there is no invariant probability distribution, while in the second case (1,0,0,...) is the 


A 
only invariant probability distribution. Finally, if P = /, the 2 x 2 identity matrix, then m = (p, 1 — p) is 
invariant for any 0 < p <1. 


Example 58.6: 


Recall that a process Xo,X1,... is called stationary if, for all positive t and s, the distribution of 
{Xo,X1,..-,Xt} is the same as the distribution of {X;,X145,...,Xt4+s}. From the definitions it follows 
that a (homogeneous) Markov chain (finite or not) is stationary if and only if j1(0) is invariant. 


A very useful tool in the calculation of invariant probability distributions is the “balance equations”: 
Ti= > Nj Pji = Tj > Pi (58.11) 
Jjri firj 


where the first equality is just a restatement of the definition of invariant probability, and the second 
follows since by Equation 58.7, the last sum equals 1. The intuition behind these equations is very useful: 
in steady state, the rate at which “probability mass enters” must be equal to the rate it “leaves.” This 
is particularly useful for continuous-time chains. More generally, given any set S, the rate at which 
“probability mass enters” the set (under the stationary distribution) equals the rate it “leaves”: 


Theorem 58.6: 


Let S be a set of states and x invariant under P. Then 


DD pH = Dom DS pa 


i€S j:j>i icS jij 
Example 58.7: 


Random walk with a reflecting barrier. This example models a discrete-time queue where, at each 
instance, either arrival or departure occurs. The state space S is the set of nonnegative integers 
(including 0), and 


Poo=1—-P, Pii41) =P, Pii-1)=1—pP_ fori>1. 


Then all states communicate so that the chain is irreducible, the chain is aperiodic and recurrent. 
From Equation 58.11 we obtain 


To = ToPoo + 11P10 
Tj = Tj—-1P(i-1yi + MWi41 P41), FS 1. 


When p < 1/2, this and Equation 58.7 imply that 1; = [(1 — 2p)/(1 — p)|pi/a — p)! for i> 0. When 
p > 1/2 the equations imply that any solution must be increasing in i and so cannot be a distribution. 
Indeed, in this case the chain is transient. 


Example 58.8: 


Birth-death process. A Markov chain on S$ = {0,1,...} is a birth-death process if pj =0 whenever 
|i —j| = 2. If X; is the number of individuals alive at time t then, at any point in time, this number can 
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increase by one (birth), decrease by one (death) or remain constant (simultaneous birth and death). 
Unlike Example 58.7, here the probability of a change in size may depend on the state. 


58.6 Random Variables 


In this section we shift our emphasis back from algebra to the stochastic process. We define some useful 
random variables associated with the Markov chain. It will be convenient to use P; for the probability 
conditioned on the process starting at state j. That is, for an event A, 


Pi(A) = P (A|Xo =i), 


with a similar convention for expectation E;. The Markov property implies that the past of a Markov 
chain is immaterial given the present. But suppose we observe a process until a random time, say the time 
a certain event occurs. Is this property preserved? The answer is positive, but only for nonanticipative 
times: 


Definition 58.7: 


Let S be a collection of states, that is, a subset of S. The hitting time ts of S is the first time the Markov chain 
visits a state in S. Formally, 
ts =inf{t > 0: X; € S}. 


Note that by convention, if X; never visits S then ts = oo. The initial time, here t = 0, does not qualify in 
testing whether the process did or did not visit S. By definition, hitting times have the following property. 
In order to decide whether or not ts = f, it suffices to know the values of Xo, ..., X+. This gives rise to the 
notion of Markov time or stopping time. 


Definition 58.8: 


A random variable t with positive integer values is called a stopping time, or a Markov time (with respect 
to the process Xo, X1,...) if one of the following equivalent conditions hold. For each t > 0 


1. It suffices to know the values of Xo, X,..., Xt in order to determine whether the event {t = t} occurred 
or not 
2. There exists a function f; so that 


Li=t() = ft (Xo(@), Sig ,X;(w)) F 


An equivalent, and more standard definition is obtained by replacing t = t by t < t. With respect to 
such times, the Markov property holds in a stronger sense. 


Theorem 58.7: Strong Markov Property 


If t is a stopping time for a homogeneous Markov chain Xo, X1,... then 


PS jin = or en eS Pe RST) 
=P(Xr=fi, jas. Xm =Julkoat 
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We can now rephrase and complement the definition of recurrence. We write t; when we really 
mean Tj}. 


Definition 58.9: 


The state j is recurrent if P;(t; < 00) = 1. It is called positive recurrent if Ejt; < 00, and null-recurrent if 


If state j is recurrent, then the hitting time of j is finite. By the strong Markov property, when the 
processes hits j for the first time, it “restarts”: therefore, it will hit j again! and again! So, let Nj be the 
number of times the process hits state j: 


ie ee) 
Nj = Ss 1x,=- 
t=1 


Theorem 58.8: 


1. Ifa state is positive recurrent, then all states in its class are positive recurrent. The same holds for null 
recurrence. 
2. Suppose jis recurrent. Then P;(N; = 00) = 1, and consequently E;N; = 00. Moreover, for every state i, 


P; (Nj = 00) = P; (tj < 00) - Pi(Nj = 00) 
=P (4; 00), 


and if P; (t; < 00) > 0 then E;N; = 00. 
3. Suppose j is transient. Then P(N; < 00) = 1, and for alli, 


E.N; = Pj (Gy < 00) 
1— P(t) <0) 


To see why the last relation should hold, note that by the strong Markov property, 


P(t; <.co anda second visit occurs) 


= Pi(tj < 00) - Pi(t; < 00), 


and similarly for later visits. This means that the distribution of the number of visits is geometric: with 
every visit we get another chance, with equal probability, to revisit. Therefore, 


E;Nj = Pi (1 < oo) +P; (1 < oo and a second visit occurs) fee 
=P; (1 < oo) (1 +P (a second visit occurs|tj < oo)) fees 
= Pi (tj < 00) (1+ Pj (ti < 00) +---) 


which is what we obtain if we expand the denominator. A similar interpretation gives rise to 


Ejt; =1+ > PikEKT. 
k:k#jeS 
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We have a simple criterion for recurrence in terms of transition probabilities, since 


CO CO 
EN SE). ty = >) Pi (X; = )= Del? 
t=1 t=1 


58.7 Limit Theorems: Transitions 


Classical limit theorems concern the behavior of t-step transition probabilities, for large t. Limits for the 
random variables are discussed in Section 58.8. 


Theorem 58.9: 


For every Markov chain, the limit 


Si Se (58.12) 


t>oo t 


exists and satisfies 


P*.P=P.P*=P*.P* = P*, 


If S is finite then P* is a stochastic matrix. 


1. Suppose the Markov chain is indecomposable, recurrent, and nonperiodic. Then, for all states i, k, 


) 
jim DO py — py | =0. 
jeS 


2. An irreducible chain is positive recurrent if and only if it has an invariant probability distribution 
uw. If it is positive recurrent and EOS then limt—oo Py = n(j) for all i,j. If it is positive 


(t+d—-1) 


recurrent with period d > 1 limt—oo ae ee =d-n(j) for alli,j. If it is null recur- 


rent then for all i,j, limt—oo o. = 0. If state j is transient then)”, Py < 00. 


Since a finite Markov chain always contains a finite closed set of states, there always exists an invariant 
distribution. Moreover, if a set is recurrent, then it is positive recurrent. 


Example 58.9: 


Example 58.3 Continued. For the periodic chain, Py ) Clearly does not converge. However, ts =1/2 
for all i,j, and the rows define an invariant iedsure 


Example 58.10: 


Example 58.8 Continued. Assume that for the birth-death process pji4.1) > 0 and pyj41); > 0 for all 
i>0Oand py > 0 for some i. Then the chain is obviously irreducible, and aperiodic (if pj = 0 for all 
i then d = 2). Using Equation 58.11 we obtain that an invariant probability distribution, if it exists, 
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must satisfy 
n= Pot: + Pi-1)i To. (58.13) 
P10 ° ++ Pi(i-1) 
Therefore, any invariant probability must satisfy 1; > 0 for all i, and in particular mg > 0. So, we can 
invoke Equation 58.7 to obtain the following dichotomy. Either 


Poi - +: Pi-1i 
Z= <o, (58.14) 
ee P10°°* Pii-1 


in which case Equations 58.13 and 58.14 determine the unique invariant probability, and we conclude 
that the Markov chain is positive recurrent. Or Z = oo, in which case there is no invariant probability 
and the chain is not positive recurrent. 


In terms of the transition matrix P, if a chain is nonperiodic, indecomposable, and recurrent then the 
matrix converges (uniformly over rows) to a matrix having identical rows, which are either all zeroes 
(null-recurrent case), or equal to the invariant probability distribution. Here are the missing cases from 
Theorem 58.9. Denote the mean hitting time of state j starting at i by mj = E;1;. Clearly mj is infinite ifj 
is not positive recurrent, and we shall use the convention that a/oo = 0 whenever a is finite. 


Theorem 58.10: 


If a state j is transient then lim;— 0 Py = 0. Ifj is recurrent with period d then lim;—o0 pee = ae [fj is 
nonperiodic, this remains true with d = 1, so that (by Theorem 58.9 ) x(j) - mj = 1. 


The last statement should be intuitive: the steady state probability of visiting state j isa measure of how 
often this state is “visited,” and this is inversely proportional to the mean time between visits. The rate at 
which convergence takes place depends on the second largest eigenvalue of P. Therefore, if the Markov 
chain is finite, indecomposable, and aperiodic with invariant probability distribution 1, then 


Ip? - x; <Ro'  foralli,j 


with p < 1. This of course implies that the one dimensional distributions converge geometrically fast. On 
the other hand, the Markov structure implies that if indeed the one-dimensional distributions converge, 
then the distribution of the whole process converges: 


Theorem 58.11: 


Suppose that for all i and j we have lim}-o0 Py = 1; for some probability distribution x. Then x is an 
invariant probability distribution, and for any i, 


dim Pi (Xt = fi Xe42 jo---) = Px (X1 =f. X2 =jo...) 


where P;, is obtained by starting the process with the distribution x. In fact, the distribution of the process 
converges in the sense that all finite dimensional distributions converge to the corresponding distributions 
under Px. 
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58.8 Ergodic Theorems 


We do not expect the Markov chain to converge: since transition probabilities are homogeneous, the 
probability of leaving a given state does not change in time. However, in analogy with iid. random 
variables, there are limit theorems under the right scaling. The connection to the i.id. case comes from 
the following construction. Fix an arbitrary state j and define 


Ry =T=5 
Ry, = inf{t > Rp-1 : Xt =j, k> 1} 
Ty = Re—-Re-1, k>1. 


Theorem 58.12: 


Ifj is recurrent and the Markov chain starts at j (with probability one), then T, Tz ... is a sequence of i.i.d. 
random variables. Moreover, the random vectors 


A 
Zk = {Xr,.XR4+1> aus XR 1) 
are independent and identically distributed (in the space of sequences of variable length!). 


This is another manifestation of the fact that, once we know the Markov chain hits some state, future 
behavior is (probabilistically) determined. Fix a recurrent state i and a time t, and define 


qt = max{Ti : Tk < #. 
Denote by N;(j) the number of times in 1,2,...,s that X, =j. By Theorem 58.12, the random vari- 


ables {Nr,.+1(/) —Npr,Qj), k=1,2,...} are independent, and (except possibly for k = 1) are identically 
distributed for each j. By the law of large numbers this implies the following. 


Theorem 58.13: 


Let i be recurrent. Then starting at i (Pj a.s.) 


a N;(j) E; peu 1x,=) Tj 
too N(0) Ej; Diy Ixne = Ne 


and x is an invariant probability distribution, concentrated on the closed class containing i, that is, 


Th = se TUjPjk> > Th = 1 


jeS k:izkeS 


so that ny = 0 ifi & k. Moreover, if we start in some state j then 


i Nii) li<oo 
im =o 


t>0o t 


with P;-probability 1. 


Mii 


The last relation implies Equation 58.12: taking E; expectations, we obtain that if i is recurrent then 
P* = P(t; < 00)/mij;. From here follows a limit theorem for functions of a Markov chain. 
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Theorem 58.14: 


Ergodic Theorem. Let S be a single recurrent class, and assume x is an invariant probability distribution. 
Let f and g be functions such that Ex If (Xo)| <0o and Ex |g(Xo)| < oo. Then for an arbitrary starting 
state i, with probability one, 


li ya) E; PE Ce) Exf (Xo) 
am t ~~ Ti — 
Bae ae g(Xs) Ej eS g(Xs) Exg(Xo) 


provided not both numerator and denominator of the last terms are zero. 


Setting g = 1 we obtain a law of large numbers for a function of the Markov chain. Here is a statement 
of a Central Limit theorem and a Law of Iterated Logarithm. 


Theorem 58.15: 


Let S be a single positive recurrent class with an invariant probability distribution n. Fix a function f and 
suppose that for some i, E; [t?] < 00 and Ex f (Xo) = 0 and 


TF 2 
E; (> i) < 00. 
t=1 
Define 
aa; 2 
YS nj-E; (x fos) 
t=1 


Then y* < 00, and if y* > 0 then 


t 
=o f (X. 
Central Limit Theorem Lszof Xs) 
/t.y2 
ty 
converges in distribution (as t > oo) to a standard Gaussian random variable. Moreover, the sum satisfies 
the Law of Iterated Logarithm, that is, the lim sup of 


Ds=oS (Xs) 
V/2y*tloglogt’ 


Law of Iterated Logarithm 


as t —> 00, is 1, and the lim inf is (—1). 


58.9 Extensions and Comments 


Markov processes are very general objects, of which our treatment covered just a fraction. Many of the 
basic ideas extend, but definitely not all, and usually some effort is required. In addition, the mathematical 
difficulties rise exponentially fast. There are two obvious directions to extend: more general state spaces, 
and continuous time. 

When the state space is not countable, the probability that an arbitrary point in the state space is 
“visited” by the process is usually zero. Therefore, the notions of communication, hitting, recurrence, and 
periodicity have to be modified. The most extensive reference here is [2]. 


Discrete Time Markov Processes 58-15 


In the discrete-space continuous time setting, the Markov property implies that if x; = i then the values 
of x;, s < t are not relevant. In particular, the length of time from the last change in state until t should 
be irrelevant. This implies that the distribution of the time between jumps (= change in state) should 
be exponential. If the only possible transition is then from i to i+ 1 and if all transition times have the 
same distribution (i.e., they are all exponential with the same parameter), then we obtain the Poisson 
process. More generally, we can describe most discrete state, continuous time Markov chains as follows. 
The process stays at state i an exponential amount of time with parameter (i). It then jumps to the 
next state according to a transition probability pj, and the procedure is repeated (this is correct if, for 
example, \(i) > > 0 for all i). This subject is covered, for example in [1] and in a new section in [2]. If 
we observe such a process at jump times, then we recover a Markov chain. This is one of the major tools 
in the analysis of continuous time chains. 

Semi-Markov processes are a further generalization, where the time between events is drawn from 
a general distribution, which depends on the state, and possibly on the next state. This is no longer a 
Markov process; however, if we observe the process only at jump times, then we recover a Markov chain. 

Finally, in applications, the information structure, and consequently the set of events, is richer: we can 
measure more than the values of the Markov chain. This is often manifested in a recursion of the type 
(Equation 58.2), but where the V; are not independent. Do we still get a Markov chain? And in what 
sense? The rough answer is that, if the Markov property (Equation 58.4) holds, but where we condition 
on all the available information, then we are back on track: all of our results continue to hold. For this to 
happen we need the “noise sequence” Vo, Vj,... to be nonanticipative in a probabilistic sense. 

Criteria for Stability: As in the case of dynamical systems, there are criteria for stability and for recur- 
rence, based on Lyapunov functions. This is one of the main tools in [2]. These techniques are often the 
easiest and the most powerful. 

Relevance to Control: Many models of control systems subject to noise can be modeled as Markov 
processes, and the discrete-time, discrete-space models are controlled Markov chains. Here the transition 
probabilities are parameterized by the control: see the section on Dynamic Programming. In addition, 
many filtering and identification algorithms give rise to Markov chains (usually with values in R®). Limit 
theorems for Markov chains can then be used to analyze the limiting properties of these algorithms. See, 
for example [2,4]. 


Example 58.11: 


Extending Example 59.2, consider the recursion 


A 
Xea1 =X t+ Ve t+ Ut 


where U; is a control variable. This is a simple instance of a controlled recursion of the ARMA type. 
Suppose that U; can only take the values +1. Of course, we require that the control depends only 
on past information. If the control values U; depend of the past states, then Xq, X1,... may not be 
a Markov chain. For example, if we choose Ut = sign (Xo), then the sequence Xg, X1,... violates the 
Markov property (Definition 59.1). However, we do have a controlled Markov chain. This means that 
Definition 59.1 is replaced with the relation 


P(X: = f|Xp-1 = it-1,--- Xo = lo, Ut_-1,.-., Uo = uo) 
= PUXt = f\Xt—1 = bt—1, Ut—1 = Ur—-1). 


This in fact is the general definition of a controlled Markov chain. If we choose a feedback control, 
that is, Up = f(X;) for some function f, then Xo, X;,. . . is again a Markov chain; but the transitions and 
the limit behavior now depend on the choice of f. 


If we are interested in optimizing a functional of a controlled chain, we obtain a Markov Decision 
process. The evaluation of the long-time average cost is often that of functionals as in Theorem 58.14. 
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Such functionals also appear in learning theory, and for the same reason. For more information on 
controlled Markov chains, see the section on Dynamic Programming. 

Finally, Markov chains have found important applications recently in the context of simulation— 
specifically in Markov Chain Monte Carlo simulation. 
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59.1 Introduction 


This chapter deals with nonlinear differential equations of the form 


dX; 


dt = a(t, Xt) + b(t, X)Zt; Xty = a, 


where Z; is a Gaussian white noise driving term that is independent of the random initial state &. Since the 
solutions of these equations are random processes, we are also concerned with the probability distribution 
of the solution process {X;}. The classical example is the Langevin equation, 


dX; 

ae —UXt + BZ;, 
where jt and f are positive constants. In this linear differential equation, X; models the velocity of a free 
particle subject to frictional forces and to impulsive forces due to collisions. Here | is the coefficient of 
friction, and B = ,/2\1kT /m, where m is the mass of the particle, k is Boltzmann’s constant, and T is the 
absolute temperature [7]. As shown at the end of Section 59.5, with a suitable Gaussian initial condition, 
the solution of the Langevin equation is a Gaussian random process known as the Ornstein—-Uhlenbeck 
process. 
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The subject of stochastic differential equations is highly technical. However, to make this chapter as 
accessible as possible, the presentation is mostly on a heuristic level. On occasion, when deeper theoretical 
results are needed, the reader is referred to an appropriate text for details. Suggestions for further reading 
are given at the end of the chapter. 


59.1.1 Ordinary Differential Equations 


Consider a deterministic nonlinear system whose state at time t is x(t). In many engineering problems, it 
is reasonable to assume that x satisfies an ordinary differential equation (ODE) of the form 


a =a(t,x(t)) + b(t,x())z(t),  x(to) =, (59.1) 


where z(t) is a separately specified input signal. Note that if we integrate both sides from fg to t, we obtain 


t 


x(t) — x(to) = / [a(, x(0)) + b(6, x(8))z(6) | do. 
to 


Since x(to) = &, x(t) satisfies the integral equation 
t 
x(t) = 8+ i [a(6, x(0)) + b(0,x(0))z(0)] 40. 
to 


Under certain technical conditions, e.g., [3], it can be shown that there exists a unique solution to 
Equation 59.1; this is usually accomplished by solving the corresponding integral equation. 

Now suppose x(t) satisfies Equation 59.1. If x(t) is passed through a nonlinearity, say y(t) := g(x(t)), 
then y(to) = g(&), and by the chain rule, y(t) satisfies the differential equation, 


dy(t) dx(t) 


Fra Sa rr (59.2) 
= g’ (x(t)) a(t, x(t) +g’ (x(t)) b(t, x(t))z(t), 


assuming g is differentiable. 


59.1.2 Stochastic Differential Equations 


If x models a mechanical system subject to significant vibration, or ifx models an electronic system subject 
to significant thermal noise, it makes sense to regard z(t) as a stochastic, or random process, which we 
denote by Z;. (Our convention is to denote deterministic functions by lowercase letters with arguments in 
parentheses and to denote random functions by uppercase letters with subscript arguments.) Now, with 
a random input signal Z;, the ODE of Equation 59.1 becomes the stochastic differential equation (SDE), 


dx 

ae =a(t,X:)+bt,X)Z1, X=, (59.3) 
where the initial condition & is also random. As our notation indicates, the solution of an SDE is a random 
process. Typically, we take Z; to be a white noise process; i.e, 


E[Z;]=0 and E[Z;Z;] = 8(t—s), 


where E denotes expectation and 8 is the Dirac delta function. In this discussion we further restrict 
attention to Gaussian white noise. The surprising thing about white noise is that it cannot exist as an 
ordinary random process (though it does exist as a generalized process [1]). Fortunately, there is a 
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well-defined ordinary random process, known as the Wiener process (also known as Brownian motion), 
denoted by W;, that makes a good model for integrated white noise, i.e., Wt behaves as if 


t 
Wi = if Zo do, 
0 
or symbolically, dW; = Z;dt. Thus, if we multiply Equation 59.3 by dt, and write dW; for Z;dt, we obtain 
aX; = a(t, Xt) dt + b(t, Xz) dw, Xtp =. (59.4) 


To give meaning to Equation 59.4 and to solve Equation 59.4, we will always understand it as shorthand 
for the corresponding integral equation, 


t t 
core i a(0,X») d+ i b(0, Xo) dWo. (59.5) 
to to 


In order to make sense of Equation 59.5, we have to assign a meaning to integrals with respect to a Wiener 
process. There are two different ways to do this. One is due to It6, and the other is due to Stratonovich. 
Since the Ité integral is more popular, and since the Stratonovich integral can be expressed in terms of 
the Ité integral [1], we restrict attention in our discussion to the Ité integral. 

Now suppose X; is a solution to the SDE of Equation 59.4, and suppose we pass X; through a nonlin- 
earity, say Y; := g(X;). Of course, Y;, = g(&), but astonishingly, by the stochastic chain rule, the analog 
of Equation 59.2 is [1] 


1 
dY, = g' (Xt) dX + <g"(X;)b(t, Xr)? dt 
2 
7 (59.6) 
= g’ (X;)a(t, Xt) dt +g (X;)b(t, Xz) dW;+ 58 (Xi), Xi)” dt, 


assuming g is twice continuously differentiable. Equation 59.6 is known as Itd’s rule, and the last term in 
Equation 59.6 is called the Ité correction term. In addition to explaining its presence, the remainder of 
our discussion is as follows. Section 59.2 introduces the Wiener process as a model for integrated white 
noise. In Section 59.3, integration with respect to the Wiener process is defined, and a simple form of 
Itd’s rule is derived. Section 59.4 focuses on SDEs. It6’s rule is derived for time-invariant nonlinearities, 
and its extension to time-varying nonlinearities is also given. In Section 59.5, It6’s rule is used to solve 
special forms of linear SDEs. ODEs are derived for the mean and variance of the solution in this case. 
When the initial condition is also Gaussian, the solution to the linear SDE is a Gaussian process, and 
its distribution is completely determined by its mean and variance. Nonlinear SDEs are considered 
in Section 59.6. The solutions of nonlinear SDEs are non-Gaussian Markov processes. In this case, 
we characterize their transition distribution in terms of the Kolmogorov forward (Fokker-Planck) and 
backward partial differential equations. 


59.2 White Noise and the Wiener Process 
A random process {Z;} is said to be a white noise process if 
E[Z;]=0 and E[Z;Z,] = &(t-—s), (59.7) 


where 8(t) is the Dirac delta, which is characterized by the two properties 3(t)=0 for t #0 and 
fo Oger, 
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Consider the integrated white noise 


t 
Wi= i Z, du. (59.8) 
0 
We show that integrated white noise satisfies the following five properties. For0 <@<t<s<t, 
Wo = 0, (59.9) 
E[ W;:] = 0, (59.10) 
E[(W; — W,)?] =t—s, (59.11) 
E[(W; — W.)(Wx — We)] = 0, (59.12) 
E[W; W,] = min{t, s}. (59.13) 


In other words: 


e Wo isa constant random variable with value zero. 

e W; has zero mean. 

« W;-— W, has variance t — s. 

« If(0,t] and (s, t] are nonoverlapping time intervals, then the increments W, — We and W; — W; 
are uncorrelated. 

e The correlation between W; and W, is E[W;W,] = min{t, s}. 


(A process that satisfies Equation 59.12 is said to have orthogonal increments. A very accessible introduc- 
tion to orthogonal increments processes can be found in [4].) 

The property defined in Equation 59.9 is immediate from Equation 59.8. To establish Equation 59.10, 
write E[W;] = is E[Z,,] du = 0. To derive Equation 59.11, write 


E[(W: - w.] = e( (2 au) ([% av)| 
=f (fee au) 
= [( [uv du) av 
= [id 


=t-s. 


To obtain Equation 59.12, write 


E[(W; — W;)(W, — Woe)] = e| ([z au)( fz, w)| = [(f-» au) av 


Because the ranges of integration, which are understood as (6, t] and (s, t], do not intersect, 8(u — v) = 0, 
and the inner integral is zero. Finally, the properties defined in Equations 59.9, 59.11, and 59.12 yield 
Equation 59.13 by writing, when t > s, 
E[W; W,] = El(W; — W.) Ws] + ELW] 
= E[(W; wa W;) Ws] +S 
= E[(W; — W,)(W; — Wo)] +5 


=S. 


(59.14) 
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If t<s, a symmetric argument yields E[W;W,] = t. Hence, we can in general write E[W;W,] = 
min{t, s}. 

It is well known that no process {Z;} satisfying Equation 59.7 can exist in the usual sense [1]. Hence, 
defining W; by Equation 59.8 does not make sense. Fortunately, it is possible to define a random process 
{W;, t > 0} satisfying Equations 59.9 through 59.13 (as well as additional properties). 


Definition 59.1: 


The standard Wiener process, or Brownian motion, denoted by {W;, t > 0}, is characterized by the follow- 
ing four properties: 


W-1 Wo = 0. 
W-2 For0 <s < t, the increment W; — W, is a Gaussian random variable with zero mean and variance 
t—s, ie., 


x2 


w os arr ‘ 
-o /2n(t—s) 


W-3 {W:,t > 0} has independent increments; i.e., if0 < ty <--+ < ty, then the increments 


Pr(W; _ WwW; < w) = 


(Wi, — Wi, )s (Wiz — Wio)s- + +> (We, — Wi, 1) 


are statistically independent random variables. 
W-4 {W;,t > 0} has continuous sample paths with probability 1. 


A proof of the existence of the Wiener process is given, for example, in [2] and in [4]. A sample path 
of a standard Wiener process is shown in Figure 59.1. 


FIGURE 59.1 Sample path of a standard Wiener process. 
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We now show that properties W-1, W-2, and W-3 are sufficient to prove that the Wiener process 
satisfies Equations 59.9 through 59.13. Clearly, property W-1 and Equation 59.9 are the same. To establish 
Equation 59.10, put s = 0 in property W-2 and use property W-1. It is clear that property W-2 implies 
Equation 59.11. Also, Equation 59.12 is an immediate consequence of properties W-3 and W-2. Finally, 
since the Wiener process satisfies Equations 59.9, 59.11, and 59.12, the derivation in Equation 59.14 holds 
for the Wiener process, and thus Equation 59.13 also holds for the Wiener process. 


Remark 59.1 


From Figure 59.1, we see that the Wiener process has very jagged sample paths. In fact, if we zoom 
in on any subinterval, say [2,4] as shown in Figure 59.2, the sample path looks just as jagged. In other 
words, the Wiener process is continuous, but seems to have corners everywhere. In fact, it can be shown 
mathematically [2] that the sample paths of the Wiener process are nowhere differentiable. In other 
words, W; cannot be the integral of any reasonable function, which is consistent with our earlier claim 
that continuous-time white noise cannot exist as an ordinary random process. 


59.3 The It6 Integral 


Let {W;, t > 0} be a standard Wiener process. The history of the process up to (and including) time tf is 
denoted by F; := o(We,0 < 9 < t). For our purposes, we say that a random variable X is F;-measurable 
if it is a function of the history up to time ¢; i-e., if there is a deterministic function h such that X = 
h(Wo, 0 < 6 < ft). For example, if X = W; — W; /2 then X is F;-measurable. Another example would be 
X= fis We d@; in this case, X depends on all of the variables Wa, 0 < 8 < t and is F;-measurable. We also 
borrow the following notation from probability theory. If Z is any random variable, we write 


E[Z|F;] instead of E[Z|Wo,0 <6 < f]. 


FIGURE 59.2 Closeup of sample path in Figure 59.1. 
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If Z is arbitrary but X is 7;-measurable, then [2] 
E[XZ|F;] = X E[Z|F;]. (59.15) 


A special case of Equation 59.15 is obtained if Z = 1; then E[Z|F,] = E[1|7;] = 1, and hence, if X is 
F;-measurable, 
E[X|F;] = xX. (59.16) 


We also require the following results from probability theory [2], which we refer to as the smoothing 
properties of conditional expectation: 


EIZ)=E[EIZ|FJ], t>0, (59.17) 


and 
E[Z|F.] =E[E(Z|F;]|F.], t>s>0. (59.18) 


Definition 59.2: 


A random process {H;,t > 0} is {F;}-adapted (or simply adapted if {F;} is understood), if for each t, Hy is 
an F,-measurable random variable. 
Obviously, {W;, t > 0} is {F;}-adapted. We now show that W; is a martingale, i.e., 
E[W:lFs]= Ws, t2s. (59.19) 
To see this, first note that since Wo = 0, 
F; := 0(Wo,0 <9 <s)=0(Wo — Wo, 0 <9 <5). 


It follows, on account of the independent increments of the Wiener process, that W; — W, is independent 
of the history F; for t > s; ie., for any function f, 


Elf (Wi — Ws)|Fs] = ELf(W: — Ws)]. 


Then, since W; — W, has zero mean, 


E[W; — Ws|F;] = E[W;-W;]=0, t>s, (59.20) 


which is equivalent to Equation 59.19. 
Having shown that W; is a martingale, we now show that W? — t is also a martingale. To do this, we 
need the following three facts: 


1. W; — W; is independent of F, with E[(W; — W,)’"]=t—s 
2. W; isa martingale (cf. Equation 59.19) 
3. Properties defined in Equations 59.15 and 59.16 


For t > s, write 
ELW;| Fs] = El(W; — W,)? + 2WiW, — We | Fi] 
= E[(W; — W,)"| Fe] + 2WSELWi|Fe] — W? 
=t—s+2W;-Wwe 
=t+W?-s. 
Rearranging, we have 
E[W?-t|\F.]=We—-s, t>s, 


ie., W? —t isa martingale. 
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59.3.1 Definition of It6’s Stochastic Integral 


We now define the It6 integral. As in the development of the Riemann integral, we begin by defining the 
integral for functions that are piecewise constant in time; i.e., we consider integrands {H;, t > 0} that are 
{F;}-adapted processes satisfying 

Ay = Hi; for tj <t < ti44, (59.21) 


for some breakpoints fj < ti;. Thus, while H; = H;, on [fj, ti+1), the value H;, is an F;,-measurable 


random variable. Without loss of generality, given any 0 < s < t, we may assume s=f9 <---<t, =t. 
Then the Ité integral is defined to be 


n—1 


t 
/ Ho dWo := Y— Ai, (Wi,, — Wa). (59.22) 
: i=0 


To handle the general case, suppose H; is a process for which there exists a sequence of processes Hk of 
the form of Equation 59.21 (where now n and the breakpoints {t;} depend on k) such that 


t 
lim | E(|Hk — Ho|?]d9=0. 
k>oo Js 
Then we take fe He dW6 to be the mean-square limit (which exists) of fe Hk dW»; i.e., there exists a 
random variable, denoted by Me ‘ Ho dWe, such that 


t t 
lim e(| Haw — [ Ho dWo 
k->0o s s 


See [10] for details. It should also be noted that when H; = h(t) is a deterministic function, the right-hand 
side of Equation 59.22 is a Gaussian random variable, and since mean-square limits of such quantities 
are also Gaussian [4], { ap : h(0) dW, t => s} is a Gaussian random process. When the integrand of an It6é 
integral is deterministic, the integral is sometimes called a Wiener integral. 


2 
=0. (59.23) 


59.3.2 Properties of the It6 Integral 


The It6 integral satisfies the following three properties. First, the It6 integral is a zero-mean random 
variable, ie., 


t 
e| | Ho aw] = 0. (59.24) 


Second, the variance of i ' Ho dW6 is 


t 2 t 
el ( Ho aw) is / E[H¢] 49. (59.25) 


Third, if X; := i. Ho dWo, then X; is a martingale, i.e., ELX;|7,] = X;, or in terms of It6 integrals, 


t 
e| | Ho dWo 
0 


We verify these properties when H; is of the form of Equation 59.21. To prove Equation 59.24, take the 
expectations of Equation 59.22 to obtain 


Ss 
r|= | HodWo, t>s. 
0 


n—1 


t 
e| | Ho aw| = SO ELH: (Wi. — We)]. 


i=0 
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By the first smoothing property of conditional expectation, the fact that H;, is F;,-measurable, and the 
properties defined in Equations 59.15 and 59.20, 


ELH: (Wi, — Wi) = E[ECH: (Wi... ~ Wi )IFa] 
= E[H,,ElW:,., = WiFi] 
=0. 


To establish Equation 59.25, use Equation 59.22 to write 


t 2 n—1n-1 
e(({ Ho a) = 0S ELA Hy (Wiis — Wi)(Wayr — Wa) 


i=0 j=0 


First consider a typical term in the double sum for which j = i. Then by the first smoothing property and 
Equation 59.15, 


ELH? (Wiis, = Wi)" = E[ELH; (Wi... ~ Wi) Fal] 
= EH; El(W:,,, = Wi)" |Ful]- 


Since W;,,, — W;, is independent of F;,, 
El(Wois — We) Fed = El(Weigs — Wes) = tit — fi 


Hence, 
ELH? (Wiis, = Wi) = ElHr \(ti+1 — ti). 


For the terms with j 4 i, we can, without loss of generality, take j < i. In this case, 


ELAy, Hi (W%;,; 4 Wi )(Wes — W;,)] = E(ELH:, Hi (Wi,.1 ~~ Wi )(Wes c— Wi )|Fi]] 
—_ El Hy, Hy (We,4. = W,, JEW: a Wil Fe] 
= 0, by Equation 59.20. 


Thus, 


t 2. n—1 t 
e[([ mvawo) |= Seti —m = ff etHga 
s i=0 7 


To show that X; := if Ho dWo is a martingale, it suffices to prove that ELX; — X,|F;] = 0. Write 


n—1 


t 
ELX; — X,|F.] = e| | Hy dWo F.| = E| Do Hi We — Wi) *.| 
: i=0 
n—1 
= 0 ELH (Ways — We )IFod. 
i=0 


Then by the second smoothing property of conditional expectation and the properties defined in Equa- 
tions 59.15 and 59.20, 


n—1 
ELX: — Xs|Fol= >> ELH, E(Wi., — WelFe]|Fo] = 0. 
i=0 
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59.3.3 A Simple Form of It6’s Rule 


The following result is essential to derive It6’s rule [1]. 


Lemma 59.1: 


For any partition of [s, t], says = to <-++ <ty=t, put A:= maxo<j<y-1 |ti41 — tl. If 


n—l 
Vi= (Wi — Wi), (59.26) 
i=0 


then 
E[|V —(t—s)|?] < 2A(t—s). 


The importance of the lemma is that it implies that as the A of the partition becomes small, the sum of 
squared increments V converges in mean square to the length of the interval, t — s. 


Proof 59.1. The first step is to note that tf — s = = (ti+1 — tj). Next, let D denote the difference 


n—1 
D:=V=(t-5) =) {Way — Wa)’ - Gi — tf 
i=0 


Thus, D is a sum of independent, zero-mean random variables. It follows that the expectation of the cross 
terms in D? vanishes, thus leaving 


EID} = TE[[ Waar — Wad? Cor — 29] 
i=0 


Put Z; = (Wi, — Wi) //ti41 — tj. Then Z; is a zero-mean Gaussian random variable with variance 1 
(which implies EZ] = 3). We can now write 


E'= Sel [22 — nein - 1) 
i=0 
n—-1 


= El(Z? —1) Mtn — ti)? 


i=0 


n—1 

= y Gs — ti)? 
i=0 
n—1 


< Yo 2A (tit — t)) =2A(t—s). 
i=0 


Let {H;, t > 0} be a continuous {F;}-adapted process. It can be shown [1] that as the partition becomes 
finer, 


n—1 n—1 t 
YO Hi (Weis — Wi)? > 0 Ay (tins — ti) > / Ho do. 
i=0 i=0 2: 


We now derive a special case of It6’s rule. Let g(w) be a twice continuously differentiable function of w. 
If Y; = g(W;), our intuition about the chain rule might lead us to write dY; = g’(W;) dW;. As we now 
show, the correct answer is dY; = g/(W1) dWi + 52" (Wi) dt. 
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Consider the Taylor expansion, 


g(ws)—g(wi) = g(wi)(we — w+ 58 On)bor — my. 


Suppose Y; = g(W;). For s = to <--- < t, = t, write 


n—1 


¥:—Ys= 0 g(Wis1) —g(We) 
i=0 


n—1 
1 
x Ye’ Wi = W:;) + 58 (Wi Weiss . wi. 

i=0 

Note that g’(W;,) and g”(W;,) are F;,-measurable. Hence, as the partition becomes finer, we obtain 
t 1 t 
Y; —Y;5 =[ g'(Wo) dWe+ 5 / g(Wo) dé. 
Ss Ss 
Writing this in differential form, we have a special case of It6’s rule: If Y; = g(W;), then 
1 
dy; = 9'(W) dw: + 58 (Wr) dt. 


Example 59.1: 


As a simple application of this result, we show that 


t w2—t 
[ Wo dWo = ‘ : ) 
0 2 


Take g(w) = w2. Then g’(w) = 2w, and g’(w) = 2. The special case of It6’s rule gives 
dY; = 2W; dW; + 1 dt. 


Converting this to integral form and noting that Yo = we =0, 


t t 
Y= Yo+ f 2W% dina +f 1d6 
0 0 
t 
= 2[ Wo dWp +t. 
0 
Since Y¥¢ = g(Wt) = w?, the result follows. As noted earlier, integrals with respect to the Wiener 


process are martingales. Since le We dWe = (w2 — t)/2, we now have an alternative proof to the one 
following Equation 59.20 that w? —tisa martingale. 
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59.4 Stochastic Differential Equations and It6’s Rule 


Suppose X; satisfies the SDE 
dX; = a(t, Xt) dt + b(t, Xz) dw: 


or equivalently, the integral equation, 
t t 
X; =X; +f a(0, X9) dO +f b(0, Xo) dWo. 
If Y; = g(X;), where g is twice continuously differentiable, we show that Y; satisfies the SDE 
1 
AY; = g'(X;) dX; + 58 (Xr) Xi)” dt. 


Using the Taylor expansion of g as we did in the preceding section, write 
n—1 l 
Yt i! Y; x concn — X;,) + 58 KiKi -X,)"I. 
i=0 


From Equation 59.28 we have the approximation 


Xt — Xt 


i x ati, Xt;)(ti41 _ ti) + b(ti, Xt) (Wi = W;,)- 
Hence, as the partition becomes finer, 


n—1 


~ gw (Xt, MX — Xt; ) 


i=0 
converges to 
[ ee00.x.) 40+ f ¢ 0.x0) aH, 
s s 
It remains to consider sums of the form (cf. Equation 59.26) 


n—1 


ps go (X; MXti44 — Xt; ye 


i=0 


The ith term in the sum is approximately 


(59.27) 


(59.28) 


(59.29) 


g" (Xx) {alti Xi) aga — 8)? + 2al ts Xe OCG Xp Mtr — H)(Wrags — Was) +b Xi)’ Weis — Wa) 


Now with A = maxo<i<n—1 |ti+1 — til, 


n—1 n—1 
Soo" (Xp altir Xe) (tin —t)| < AD 1g” (X, lati, Xs)" (tiga — ti), 
i=0 i=0 


which converges to zero as A — 0. Also, 


n—1 


Soo" (Xr alti. Xt )(tin — ti) (Wiis — Wes) 
i=0 
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converges to 0. Finally, note that 


n—1 
So" (Xt )b(ti Xt)? (Wri — Wa)? 
i=0 


converges to ft g" (X9)b(0, Xq)? dO. Putting this all together, as the partition becomes finer, we have 


t 1 t 
7 eo / g’ (Xo) [a(0, Xo) dt + b(0, Xo) dWo| + ; / @”(Xo)b(0, Xo)? d0, 


which is indeed the integral form of It6’s rule in Equation 59.29. 
Itd’s rule can be extended to handle a time-varying nonlinearity g(t, x) whose partial derivatives 


ag 
(9 ae as d LK Ry 
gt 8 iP Pace Ox? 


are continuous [1]. If Y; = g(t, X;), where X; satisfies Equation 59.27, then 
AY; = gr(t, Xp) dt + ge(t, Xt) AX + 4 exx(t,Xp)O(t, Xr) dt 


1 (59.30) 
= g(t, Xt) dt + gy(t, Xp)a(t, Xp) dt + g(t, X¢)b(t, Xt) dW; + 3 Sux (ts X¢)B(t, Xt)” dt. 


Example 59.2: 


Consider the Langevin equation 
dX; = —3X; dt +5 dW. 


Suppose Y; = sin(tX;). Then with g(t, x) = sin(tx), gz(t, x) = x cos(tx), gx (t, x) = t cos(tx), and gxx(t, x) = 
—t? sin(tx). By the extended It6’s rule, 


25 
dY; =[(1 — 3t)X; cos(tX;) — st sin(tX;)] dt + 5t cos(tX+) dW}. 


59.5 Applications of It6’s Rule to Linear SDEs 


Using It6’s rule, we can verify explicit solutions to linear SDEs. By a linear SDE we mean an equation of 
the form 


If & is independent of {W; — W;,,t > to}, and if c, B, and y are bounded on a finite interval [to, tf], then a 
unique continuous solution exists on [fo, ab if a, c, B, and y are bounded on [to, tf] for every finite tf > to, 
then a unique solution exists on [tg, 00) [1]. 


59.5.1 Homogeneous Equations 


A linear SDE of the form 
dX; = c(t)X; dt + y(t)X; dw;, Xty = 8, (59.32) 


is said to be homogeneous. In this case, we claim that the solution is X; = & exp(Y;), where 


2, 
i ‘A car ao+ [wan 
to 


to 
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or in differential form, 


t)— y(t)? 
i joo | dt + y(t) dW. 
To verify our claim, we follow [1] and simply apply It6’s rule of Equation 59.29: 
1 
dX; = Bexp(Y;) dY; + 5 E exp(Y;)y(t)* dt 


t) — y(t)” 1 
anys eae dt + Xey(t) dWi + 5Xey(@) dt 
= c(t)X;¢ dt + y(t) Xt dw:. 


Since X;, = E exp(Y;,) = & exp(0) = G, we have indeed solved Equation 59.32. 


Example 59.3: 


Consider the homogeneous SDE 
dX; = cos(t)X; dt + 2X;dW;, Xo =1. 


For this problem, Yt = sin(t) — 2t + 2W;, and the solution is X = e’t = esin()—2(t—- WM) 


59.5.2 Linear SDEs in the Narrow Sense 
A linear SDE of the form 


is said to be linear in the narrow sense because it is obtained by setting y(t) = 0 in the general linear SDE 
in Equation 59.31. The solution of this equation is obtained as follows. First put 


O(t, fo) := exp([ a). 
to 


OP(t, to) 
Ot 
®(fo, to) = 1, and ®(E, to) (to, 8) = P(t, 0). Next, let 


Observe that 
= c(t) P(t, to), (59.34) 


t t 
Y,:= a+ [ (to, 0)a(0) ao | ®(to, 0)B(0) dWo, 


to to 
or in differential form, 
dY; = P(to, t)a(t) dt + P(to, t)B(t) dW. 
Now put 


Xt = O(t, to) Yt 
t t (59.35) 
= A(t, fy) +f @(t, 0)a(8) d6+ / @(t, 9)B(8) dWo. 
to to 
In other words, X; = g(t, Y¢), where g(t, y) = P(t, to) y. Using Equation 59.34, g(t, y) = c(t) P(t, to)y. We 
also have &y (ts y) = (tf, fo), and Sylty) = 0. By the extended It6’s rule of Equation 59.30, 


1 
dX: = Belts Yr) dt + Sy(t Vt) dVe + 5 y(t Yr) dt 
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= C(t) P(E, to); dt + P(t, to)[P (to, tha(t) dt + P(to, t)B(t) dWi] 
= c(t)X; dt + a(t) dt + B(t)dW;, 


which is exactly Equation 59.33. 

Recalling the text following Equation 59.23, and noting the form of the solution in Equation 59.35, we 
see that {X;} isa Gaussian process if and only if E is a Gaussian random variable. In any case, we can always 
use Equation 59.35 to derive differential equations for the mean and variance of X;. For example, put 
m(t) := E[X;]. Since Ité integrals have zero mean (recall Equation 59.24), we obtain from Equation 59.35, 


t 
m(t) = ®(f, to)E[E] +f P(t, 9)a(8) dd, 


to 
and thus 


t 
on) = c(t) ®(E, to)ELE] + P(t, t)a(t) +f c(t) ®(t, @)a(0) de 
to 


= c({ ®¢ to) E[S] +f @(t, 8)a(0) a| + a(t) 
to 


= c(t)m(t)+ a(t), 


and m(t)) = E[&]. We now turn to the covariance function of X;, r(t,s) := E[(X; — m(t))(X; — m(s))]. 
We assume that the initial condition & is independent of {W; — W;,, t > to}. Write 


t 
X; — m(t) = ®(t, to)(& — EL S]) +f @(t, 0)B(8) dWo. 


to 


For s < ft, write 


t Ss t 
/ &(t, 0)B(0) dWo = ; @(t, 0)B(0) dWo + / ®(t, 0)B(0) dWo. 


to to s 
Then : 
r(t,s) = ®(t, fo) E[(S — E[S])7]®(s, to) +f ®(t, 0)B(0)* B(s, 0) dd. 
to 
Letting var() := E[(S — E[&])*], for arbitrary s and t, we can write 
min{s,t} 


r(t,s) = B(t, to)var( EZ) P(s, to) +f O(t, 9)B(8)? B(s, 0) dé. 


to 


In particular, if we put v(t) := r(t, t) = E[(X; m(t))*], then a simple calculation shows 


dv(t) 
dt 


= 2c(t)v(t) + B(t)?, v(to) = var(&). 


59.5.3 The Langevin Equation 


If a(t) = 0 and c(t) and B(t) do not depend on f, then the narrow-sense linear SDE in Equation 59.33 
becomes 
dX; => cXt dt+ BdWi, Xty => On 


which is the Langevin equation when c < 0 and B > 0. Now, since 


t 
O(t, to) = exp( | zo) = elf], 
to 
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the solution in Equation 59.35 simplifies to 
t 
X, = ell gy i e818 dWo. 


to 


Then the mean is 
m(t) := EX] = et -SlE[s), 


and the covariance is 


min{t,s} 
r(t, 5) = tote tlyar( a) + f eclt—O+s—0] 2 do 


to 


2 2 
= eltts—2to] var(&) 4 E + etlt-sl =e ; 
2c 2c 


Now assume c < 0, and suppose that E[&] = 0 and var() = —B?/(2c). Then 


_Q2 
E[X;]=0 and roy =ett-9(E), (59.36) 


Since E[X;] does not depend on ft, and since r(t, s) depends only on |t — s|, {X;, t > to} is said to be wide- 
sense stationary. If © is also Gaussian, then {X;,t > fo} is a Gaussian process known as the Ornstein- 
Uhlenbeck process. 


59.6 Transition Probabilities for General SDEs 


The transition function for a process {X;, t > to} is defined to be the conditional cumulative distribution 
P(t, y|s,x) = Pr(X; < y|X; = x). 


When X; is the solution of an SDE, we can write down a partial differential equation that uniquely deter- 
mines the transition function if certain hypotheses are satisfied. Under these hypotheses, the conditional 
cumulative distribution P(t, y|s, x) has a density p(t, y|s, x) that is uniquely determined by another partial 
differential equation. 

To motivate the general results below, we first consider the narrow-sense linear SDE in Equation 59.33, 
whose solution is given in Equation 59.35. Since ®(t,0) = O(t,s)®(s,8), Equation 59.35 can be 
rewritten as 


t t 
X= (49) 06 me | ®(s, 6)a(8) ao | ®(s, 6)B(8) aw| 
to 


to 


- ©(1.9)[X+ / * ©(s,6)a(6) d+ ie * ©(s,6)8(0) aw| 


= onIK+ [/ O(a) a+ [06 8)B(0) dWo. 


Ss Ss 


Now consider Pr(X; < y|X; = x). Since we are conditioning on X; = x, we can replace X; in the preceding 
equation by x. For notational convenience, let 


Zi= otoet [ O6.0u0) ao+ f° 2(6,0)6(0) dW 


Then Pr(X; < y|X; = x) = Pr(Z < y|X; = x). Now, in the definition of Z, the only randomness comes 
from the It6 integral with deterministic integrand over [s, t]. The randomness in this integral comes only 
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from the increments of the Wiener process on [s, t]. From Equation 59.35 with t replaced by s, we see that 
the only randomness in X, comes from & and from the increments of the Wiener process on [fg, s]. Hence, 
Z and X, are independent, and we can write Pr(Z < y|X, = x) = Pr(Z < y). Next, from the development 
of the Ité integral in Section 59.3, we see that Z is a Gaussian random variable with mean 


t 
m(t|s,x) := ons) [ @(t, 8)a(8) dO 


o: 
and variance 


t 
v(t|s) := / [H(t, 0)B(A)]* dd. 


Hence, the transition function is 


y exp Gc —m(t|s, s9P /ut)) 
P(t, yls,x) = 
ues ie / 21v(t|s) 


and the transition density is 


dz, 


exp (50 - mls? /o(l)) 


pt yls,x) = 


Example 59.4: 


Consider the SDE dX; = —3X; dt + 5 dW. Then m(t|s,x) = e~3)x, and v(t|s) = Bf1 — e~ O-9), 


We now return to the general SDE, 


dX; = a(t, Xt) dt + b(t, X;) dw, Xty = 6, (59.37) 


Lo 


where & is independent of {W; — W;,,,t => to}. To guarantee a unique continuous solution on a finite 
interval, say [to, t¢], we assume [1] that there exists a finite constant K > 0 such that for all t € [to, tf] and 
all x and y, a and b satisfy the Lipschitz conditions 


|a(t, x) a a(t, y)| < K|x —y\ 
|b(t, x) — b(t, y)| < K|x — yl, 


and the growth restrictions 


|a(t, x)|? < K7(14 |x|), 
[b(t,x)|? < K7(1 + |x|”). 


If such a K exists for every finite tf > to, then a unique solution exists on [tp, 00) [1]. Under the above 
conditions for the general SDE in Equation 59.37, a unique solution exists, although one cannot usually 
give an explicit formula for it, and so one cannot find the transition function and density as we did in 
the narrow-sense linear case. However, if for some bo > 0, b(t, x) => bo for all ¢ and all x, then [10] the 
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transition function P is the unique solution of Kolmogorov’s backward equation 


OP(t, y|s, x) i OP(t, y|s, x) 


2 
1 ah O P(t, y|s, x) 


b ; sot t<t, 59.38 
5 D2 + a(s, x) Ox me o<s<t<f (59.38) 
satisfying 

: Ji, y>x, 

He P(t, y|s, x) = to ee 
Furthermore, P has a density, 

OP(t, yls, 
H(bsy}8) = FEE, ty<s<t<t. 


If 0a/Ox, 0b/Ax, and 07b/ Ax? also satisfy the Lipschitz and growth conditions above, and if 0b/Ox > bo > 
0 and 07b/Ax” > bo > 0, then the transition density p is the unique fundamental solution of Kolmogorov’s 
forward equation 


1 &[b(t,y)’p(t.yls,x)] Oat, y)p(t,yls,x)] _ Op(t, yls, x) 
2 ay? dy 7 ot 


ty<s<t<ft, (59.39) 


satisfying 
Pls, y|s, x) = &(y — x). 
The forward partial differential equation is also known as the Fokker—Planck equation. Equation 59.39 is 


called the forward equation because it evolves forward in time starting at s; note also that x is fixed and y 
varies. In the backward equation, t and y are fixed, and x varies as s evolves backward in time starting at f. 


Remark 59.2 


If the necessary partial derivatives are continuous, then we can differentiate the backward equation with 
respect to y and obtain the following equation for the transition density p: 


2 
O° p(t, yls,x) ple eS) e Op(t, y|s, x) 


1 
U(s, ; 
2 (s.4) Ox? Ox Os 


flo<s<t<f. 


Example 59.5: 


In general, it is very hard to obtain explicit solutions to the forward or backward equations. However, 
when a(t, x) = a(x) and b(t, x) = b(x) do not depend on t, it is sometimes possible to obtain a limiting 
density p(y) = limt— oo p(t, y|s, x) that is independent of s and x. The existence of this limit suggests 
that for large t, p(t, y|s, x) settles down to a constant as a function of t. Hence, the partial derivative 
with respect to t on the right-hand side of the forward equation in Equation 59.39 should be zero. 
This results in the ODE 

1 d*[by)*p)]  dlaly)p(y)] _ 

2 dy2 dy 


For example, let j and be positive constants, and suppose that 


AX, = — Xe dt + / (1 +X?) dW. 


(The case » = 1 and ) = 2 was considered by [10].) Then Equation 59.40 becomes 


0. (59.40) 


1 d?[A(1+y7)ply)]  di-pypy)] 
2 dy2 dy 


0. 
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Integrating both sides, we obtain 


» di +y2 
x [( YORU ye 


for some constant «. Now, the left-hand side of this equation is (A + w)yp(y) + 4(1 + y2)p/(y) /2. If we 
assume that this goes to zero as |y| — oo, then x = 0. In this case, 
ply) _ € =) 2y 
ply) n J ity? 


Integrating from 0 to y yields 


or 


Of course, p(0) is determined by the requirement that 4 ee p(y) dy = 1. For example, if 1/2 = 1/2, 
then p(0) = 1/2, which can be found directly after noting that the antiderivative of 1/(1 + y2)3/2 
is y/V1 +y2. As a second example, suppose 1/2 = 1. Then p(y) has the form p(0)f(y)2, where 
f(y)=1/01 +y?). If we let F(a) denote the Fourier transform of f(y), then Parseval’s equation yields 
S85 MFI? dy = [PS |F(w)|? dw/2x. Since F(w) = ne7!®!, this last integral can be computed in 
closed form, and we find that p(0) = 2/1. 


59.7 Defining Terms 


Adapted: A random process {H;} is {7;}-adapted, where F; = o(We,0 < 9 < t), if for each t, H; is F;- 
measurable. See measurable. 

Brownian motion: Synonym for Wiener process. 

Fokker-Planck equation: Another name for Kolmogorov’s forward equation. 

History: The history of a process {Wo, 6 > 0} up to and including time t is denoted by F; = 0(Wo,0 < 
6 < t). See also measurable. 

Homogeneous: Linear stochastic differential equations (SDEs) of the form in Equation 59.32 are homo- 
geneous. 

Integrated white noise: A random process W; that behaves as if it had the representation W; = /, Zo AO, 
where Zg is a white noise process. The Wiener process is an example of integrated white noise. 

It6 correction term: The last term in It6’s rule in Equations 59.6 and 59.30. This term accounts for the 
fact that the Wiener process is not differentiable. 

It6’s rule: A stochastic version of the chain rule. The general form is given in Equation 59.30. 

Kolmogorov’s backward equation: The partial differential Equation 59.38 satisfied by the transition 
function of the solution of an SDE. 

Kolmogorov’s forward equation: The partial differential Equation 59.39 satisfied by the transition den- 
sity of the solution of an SDE. 

Martingale: {W;} is an {F;}-martingale if {W;} is {F;}-adapted and if for all t > s > 0, E[W:|F;] = Ws, 
or equivalently, E[W; — W,|F;] = 0. 

Measurable: See also history. Let F; = o(Wo, 0 < 0 < t). A random variable X is ¥;-measurable if it is a 
deterministic function of the random variables {Wo, 0 < 0 < ft}. 

Narrow sense: An SDE is linear in the narrow sense if it has the form of Equation 59.33. 

Ornstein-Uhlenbeck process: A Gaussian process with zero mean and covariance function in 
Equation 59.36. 
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Smoothing properties of conditional expectation: See Equations 59.17 and 59.18. 

Transition function: For a process {X;}, the transition function is P(t, y|s, x) := Pr(X; < y|X; = x). 

White noise process: A random process with zero mean and covariance E[Z;Z;] = 8(t — s). 

Wiener integral: An Ité integral with deterministic integrand. Always yields a Gaussian process. 

Wiener process: A random process satisfying properties W-1 through W-4 in Section 59.2. It serves as a 
model for integrated white noise. 
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Further Reading 


For background on probability theory, especially conditional expectation, we recommend [2]. 

For linear SDEs driven by orthogonal-increments processes, we recommend the very readable text by 
Davis [4]. 

For SDEs driven by continuous martingales, there is the more advanced book by Karatzas and 
Shreve [7]. 

For SDEs driven by right-continuous martingales, the theory becomes considerably more complicated. 
However, the tutorial paper by Segall [9], which compares discrete-time and continuous-time results, is 
very readable. Also, Chapter 6 of [10] is accessible. 

Highly technical books on SDEs driven by right-continuous martingales include [5] and [8]. 

For the reader interested in Markov processes there is the advanced text of Ethier and Kurtz [6]. 
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60.1 Introduction 


Stationary stochastic processes are good ways of modeling random disturbances. The treatment here 
is basically for linear discrete-time input-output models. Most modern systems for control and signal 
processing work with sampled data; hence discrete-time models are of primary interest. 

Properties of models, ways to calculate variances, and other second-order moments are treated. This 
chapter is organized as follows. Autoregressive moving average (ARMA) and ARMAX (ARMA with an 
eXogenous input) models are introduced in Section 60.2, while Section 60.3 deals with the effect of linear 
filtering. Spectral factorization, which has a key role when finding appropriate model representations for 
optimal estimation and control, is described in Section 60.4. Some ways to analyze stochastic systems by 
covariance calculations are presented in Section 60.5, while Section 60.6 gives a summary of results for 
continuous-time procecces. 

In stochastic control, it is fundamental to predict future values of the process. A more general situation 
is the problem to estimate unmeasurable variables. Mean square optimal prediction is dealt with in 
Section 60.7, with minimal output variance control as a special application. In Section 60.8, a more 
general estimation problem is treated (covering optimal prediction, filtering, and smoothing), using 
Wiener filters, which are described in some detail. 

This chapter is based on [6,7], where proofs and derivations can be found, as well as several extensions 
to the multivariable case, and to complex-valued signal processing problems. Other aspects, primarily 
related to control, can be found, for example, in [1-5]. 
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60.2 ARMA and ARMAX Models 


Wide sense stationary random processes are often characterized by their first- and second-order moments, 
that is by the mean value 


m = Ex(t) (60.1) 

and the covariance function 
r(t)  E[x(¢ + t) — m][x(t) — m), (60.2) 
where f, t take integer values 0,-+1+ 2,.... For a wide sense stationary processes the expected values in 


Equations 60.1 and 60.2 are independent of t. As an alternative to the covariance function one can use its 
discrete Fourier transform, that is, the spectrum, 


(oe) 


o(z)= DS > raz". (60.3) 


n=—OCo 


Evaluated on the unit circle it is called the spectral density, 


o(e)= S> r(nje“. (60.4) 
As 
r(t)= Ss o(e!)e@ da = 22 § bee (60.5) 
2m J_x 201 Zz 


(where the last integration is counterclockwise around the unit circle) the spectral density describes how 
the energy of the signal is distributed over different frequency bands (set t = 0 in Equation 60.5). 

Similarly, the cross-covariance function between two wide sense stationary processes y(t) and x(t) is 
defined as 


ryx(T) 2 E[y(t +t) — my][x(t) — my], (60.6) 
and its associated spectrum is 
CO 
Pyx(Z) = > Pyx(n) Zz". (60.7) 
n=—0Oo 


A sequence of independent identically distributed (i.i.d.) random variables is called white noise. A white 
noise will have 


r(t)=0 fort 0. (60.8) 


Equivalently, its spectrum is constant for all z. Hence its energy is distributed evenly over all frequencies. 
In order to simplify the development here, it is generally assumed that signals have zero mean. This 
is equivalent to considering only deviations of the signals from an operating point (given by the mean 
values). 
Next, an important class of random processes, obtained by linear filtering of white noise, is introduced. 
Consider y(t) given as the solution to the difference equation 


y(t) +ayy(t — 1) +--+ + any(t —n) = e(t)+ cye(t — 1) +--+ +cme(t — m), (60.9) 


where e(t) is the white noise. Such a process is called an ARMA process. If m = 0, it is called an autore- 
gressive (AR) process, and if n = 0 a moving average (MA) process. 
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Introduce the polynomials 


A(z) = 2" +ayz" 1 4---+4n, 


. ae (60.10) 
C(z) =z" +z +++ +m, 


and the shift operator q, qx(t) = x(t + 1). The ARMA model (Equation 60.9) can then be written com- 
pactly as 


A(q)y(t — n) = C(q)e(t — m). (60.11) 


As the white noise can be “relabeled” without changing the statistical properties of y(t), (Equation 60.9) 
is much more frequently written in the form 


A(q)y(t) = C(q)e(t). (60.12) 


Some illustrations of ARMA processes are given next. 


Example 60.1: 


Consider an ARMA process 
A(q)y(t) = C(q)e(t). 


The coefficients of the A(q) and C(q) polynomials determine the properties of the process. In parti- 
cular, the roots of A(z), which are called the poles of the process, determine the frequency contents 
of the process. The closer the poles are located toward the unit circle, the slower or more oscillating 
the process. Figure 60.1 illustrates the connections between the A and C coefficients, realizations 
of processes and their second-order moments as expressed by covariance function and spectral 
density. 


As an alternative to q, one can use the backward shift operator, q7!. An ARMA model would then be 
written as 


A(q~')y(t) = C(q7")e(t), 


where the polynomials are 


A(q”')=1taiqe) +++++ ang", 
C(q"') =1+ aq? +-+-+¢mq™. 


The advantage of using the q-formalism is that stability corresponds to the “natural” condition |z| < 1. 
An advantage of the alternative q~!-formalism is that causality considerations (see below) become easier 
to handle. The q-formalism is used here. 

In some situations, such as modeling a drifting disturbance, it is appropriate to allow A(z) to have zeros 
on or even outside the unit circle. Then the process will not be wide sense stationary, but drift away. It 
should hence only be considered for a finite period of time. The special simple case A(z) = z — 1, C(z) =z 
is known as a random walk. 

If an input signal term is added to Equation 60.10, we obtain an ARMAX model 


A(q)y(t) = B(q)u(t) + C(qg)e(t). (60.13) 
The system must be causal, which means that y(t) is not allowed to depend on future values of the input 


u(t+t), t>0. Hence it is required that deg A > deg B. Otherwise y(t) would depend on future input 
values. 
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FIGURE 60.1 Illustration of some ARMA processes. (Process I: A(q) = q? — 1.5q +0.8, C(q) = 1. Pole locations: 
0.75 + i0.49. Process II: A(q) = q? — 1.0q + 0.3, C(q) = 1. Pole locations: 0.50 + i0.22. Process III: A(q) = q? — 0.5q+ 


0.8, C(q) = 1. Pole locations: 0.25 + i0.86. Process IV: A(q) = q?, C(q) = q” 


0.5q + 0.9. Pole locations: 0, 0.) 
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Sometimes higher-order moments (i.e., moments of order higher than two) are useful. Such spectra 
are useful tools for the following: 


« Extracting information due to deviations from a Gaussian distribution. 
¢ Estimating the phase of a non-Gaussian process. 
¢ Detecting and characterizing nonlinear mechanisms in time series. 


To exemplify, we consider the bispectrum, which is the simplest form of higher-order spectrum. 
Bispectra are useful only for signals that do not have a probability density function that is symmetric 
around its mean value. For signals with such a symmetry, spectra of order at least four are needed. 

Let x(t) be a scalar stationary process of zero mean. Its third moment sequence, R(m, n) is defined as 


R(m, n) = Ex(t)x(t + m)x(t +n), (60.14) 
and satisfies a number of symmetry relations. The bispectrum is 
CO [o,@) 
Bena) Y.  Y> Remn)zy™z,". (60.15) 
m=—CO N=—CO 


Let us consider two special cases: 


e Let x(t) be zero mean Gaussian. Then 
R(m,n)=0, B(z,z2) =0. (60.16) 


« Let x(t) be non-Gaussian white noise, so that x(t) and x(s) are independent for t # s, Ex(t) = 0, 
Ex?(t) = 0”, Ex?(t) =. Then 


_ fp ifm=n=0, 
Ete fe elsewhere. cont) 
The bispectrum becomes a constant, 
B(z1, 22) = B. (60.18) 
60.3 Linear Filtering 
Let u(t) be a stationary process with mean m, and spectrum $,(z), and consider 
lo) 
y(t) = Gqult) = D> geult—b), (60.19) 


k=0 


where G(q) = >-?7°.9 gq” * is an asymptotically stable filter (e.g., it has all poles strictly inside the unit 
circle). Then y(t) is a stationary process with mean 


my = G(1)my, (60.20) 
and spectrum ¢,(z) and cross-spectrum $y,(z) given by 


by(z) = G(z)G(z~")ou(z), (60.21) 
byu(Z) = G(z)bu(z). (60.22) 


The interpretation of Equation 60.20 is that the mean value m,, is multiplied by the static gain of the 
filter to obtain the output mean value my. 
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The following corollary is a form of Parseval’s relation. Assume that u(t) is a white-noise sequence with 
variance \”. Then 


2 2 = 7 1,4 
EP (t)}= > ge = a § G@GE")—, (60.23) 
k=0 


Let u(t) further have bispectrum B,,(z;, 22). Then By(z;, z2) can be found after straightforward calcula- 
tion, 
By (21, 22) = G(zy zy ')G(z1)G(22)Bu(za, 22). (60.24) 
This is a generalization of Equation 60.21. Note that the spectral density (the power spectrum) 
does not carry information about the phase properties of a filter. In contrast to this, phase prop- 
erties can be recovered from the bispectrum when it exists. This point is indirectly illustrated in 
Example 60.1. 


60.4 Spectral Factorization 


Let $(z) be a scalar spectrum that is rational in z, that is, it can be written as 


<m Buz 
o(z) = Zaks (60.25) 
ikea Kz 
Then there are two polynomials: 
A(z) =z" +ayz" 1 4-+-+4n, 
(60.26) 


C(z) =z" 4c 2 4 ACs 


and a positive real number 2 so that, (1) A(z) has all zeros inside the unit circle, (2) C(z) has all zeros 
inside or on the unit circle, and (3) 
sie) 
A(z) A(z7!) 

In the case where o(e”) > 0 for all w, C(z) will have no zeros on the circle. 

Note that any continuous spectral density can be approximated arbitrarily well by a rational function 
in z = e as in Equations 60.25 through 60.27, provided that m and n are appropriately chosen. Hence, 
the assumptions imposed are not restrictive. Instead, the results are applicable, at least with a small 
approximation error, to a very wide class of stochastic processes. 

It is an important implication that (as far as second-order moments are concerned) the underlying 
stochastic process can be regarded as generated by filtering white noise, that is, as an ARMA process 


A(q)y(t) = C(q)e(t), 
Ee*(t) = 2. 


(60.27) 


(60.28) 


Hence, for describing stochastic processes (as long as they have rational spectral densities), there is no 
restriction to assume that the input signals are white noise. In the representation (Equation 60.28), the 
sequence {e(t)} is called the output innovations. 

Spectral factorization can also be viewed as a form of aggregation of noise sources. Assume, for example, 
that an ARMA process 

A(q)x(t) = C(q)v(t) (60.29) 


is observed but the observations include measurement noise 
y(t) = x(t) + e(t), (60.30) 


and that v(t) and e(t) are uncorrelated white-noise sequences with variances )? and 22, respectively. As 
far as the second-order properties (such as the spectrum or the covariance function) are concerned, y(t) 
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FIGURE 60.2 Two representations of an ARMA process with noisy observations. 


can be viewed as generated from one single noise source: 
A(q)y(t) = D(q)e(t). (60.31) 


The polynomial D(q) and the noise variance 2 are derived as follows. The spectrum is, according to 
Equations 60.29 through 60.31, 


4 C(z)C(z“!) 2 > D(z)D(z“!) 
oy(@) = vat) e* A(z)A(z7!)" 


Equating these two expressions gives 
h2D(z)D(z~!) = NC(z)C(z7!) + X2A(z)A(z7!). (60.32) 


The two representations (Equations 60.29 and 60.30), and Equation 60.31 of the process y(t), are 
displayed schematically in Figure 60.2. 

Spectral factorization can also be performed using a state-space formalism. Then an algebraic Riccati 
equation (ARE) has to be solved. Its different solutions corresponds to different polynomials C(z) satis- 
fying Equation 60.27. The positive-definite solution of the ARE corresponds to the C(z) polynomial with 
all zeros inside the unit circle. 

One would expect that for a given process, the same type of filter representation will appear for the 
power spectrum and for the bispectrum. This is not so in general, as illustrated by the following example. 


Example 60.2: 


Consider a process consisting of the sum of two independent AR processes 


1 1 
(t) = —~e(t)+ ——~ vit), (60.33) 
we AQ Clq) 
e(t) being Gaussian white noise and v(t) non-Gaussian white noise. Both sequences are assumed to 
have unit variance, and Ev3(t) = 1. 


The Gaussian process will not contribute to the bispectrum. Further, R,(z1, Z2) = 1, and according to 
Equation 60.24 the bispectrum will be 


1 1 1 
By(Z1, 22) = 5 . 
yore) C(zz ‘zy ') C(z1) C(z2) es 
so 
1 
HO = a5 (60.35) 


is the relevant filter representation as far as the bispectrum is concerned. However, the power spectrum 
becomes 

1 ; 1 
A(e)A(e-!) * C(2)C(e“1)’ 


o(z) = (60.36) 
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and in this case it will have a spectral factor of the form 


_ BG) 
H(z) = A@C@)’ (60.37) 
where 
B(z)B(z~!) = A(z)A(z~!) + C(z)C(z7) (60.38) 


due to the spectral factorization. Clearly, the two filter representations of Equations 60.35 and 60.37 differ. 


60.5 Yule—Walker Equations and Other Algorithms 
for Covariance Calculations 


When analyzing stochastic systems it is often important to compute variances and covariances between 
inputs, outputs, and other variables. This can mostly be reduced to the problem of computing the covari- 
ance function of an ARMA process. Some ways to do this are presented in this section. 

Consider first the case of an AR process 


y(t) + ayy(t—1)+---+any(t—n) = e(t), Ee?(t) =’. (60.39) 


Note that y(t) can be viewed as a linear combination of all the old values of the noise, that is, {e(s)}‘__,. 


By multiplying y(t) by a delayed value of the process, say y(t — t), t > 0, and applying the expectation 
operator, one obtains 


Ey(t — v)[y(t) + ary(t — 1) +--+ + any(t — n)] = Ey(t — ve(t), 
or 
Re ane Hem ees ara ree | ty (60.40) 
Mw, t=0, 
which is called a Yule-Walker equation. By using Equation 60.40 for t = 0,...,, one can construct the 
following system of equations for determining the covariance elements r(0), r(1),...,7(m): 
1 ay ee an r(0) 2 
a, l+qam an O . 0 
a el ie (60.41) 
Gn GAn-| +s 1 r(n) 0 


Once r(0),...,r(m) are known, Equation 60.40 can be iterated (for t=n+1,n+2,...) to find further 
covariance elements. 
Consider next a full ARMA process 


y(t) + ayy(t — 1) +--+ + any(t —n) = e(t) + cye(t —1) +--+ +¢me(t — m), 
Ee*(t) =. (60.42) 


Now computing the cross-covariance function between y(f) and e(t) must involve an intermediate step. 
Multiplying Equation 60.42 by y(t — t), t > 0, and applying the expectation operator, gives 


ry(t) + ayy (t — 1) ++ ++ + anty(t — 1) = rey(t) + C1 rey(t — 1) ++ ++ + Cmtey(t — m). (60.43) 


In order to obtain the output covariance function ry(t), the cross-covariance function rey(t) must first 
be found. This is done by multiplying Equation 60.42 by e(t — t), and applying the expectation operator, 
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which leads to 


Tey(—t) + aj Tey( t+ 1) eae! Antey( t+ n) = [8.0 t 184-10 by ee Cmd1—m, 0] (60.44) 


where §;,; is the Kronecker delta (8+; = 1 if t = s, and 0 elsewhere). As y(t) is a linear combination of 
{e(s)}{__.> it is found that rey(t) = 0 for t > 0. Hence Equation 60.43 gives 


ty(t) + ary (t—I +++ +anry(t—n)=0, t>m. (60.45) 
The use of Equations 60.43 through 60.45 to derive the autocovariance function is illustrated next by 


applying them to a first-order ARMA process. 


Example 60.3: 


Consider the ARMA process 
y(t) +ay(t—1)=e(t)+ce(t—1), Fe2(t) = 22. 

In this case, n = 1,m = 1. Equation 60.45 gives 

ry(t)+ar(t—1)=0, t>1. 
Using Equation 60.43 for t = 0 and 1 gives 

¢ ‘) a) = (: 5) ( rey(0) ) 
a 1) \ry()} \e OF \tey(-1))° 

Consider Equation 60.44 for t = 0 and 1, which gives 

fe Glewee(o): 
By straightforward calculations, it is found that 

rey(0)=27, rey(-1) =9*(c—a), 


2 n2 
(1 +c? —2ac), ry()= ; 


ry(0) = 
‘y (0) fw 


and finally, 
2 
1—a2 


ry(t) = (c—a)(1—ac)(-a)"""', t>1. 


As an example of alternative approaches for covariance calculations, consider the following situation. 
Assume that two ARMA processes are given: 


A(q)y(t) = Biq)e(t), 
A(q) = q" +aiq™ +--+ +4n; (60.46) 
B(q) = boq" + bq” | +--+ +bn 
and 
C(q)w(t) = D(g)e(t), 
C(q)=q" + rt ae Oe Seale ee (60.47) 
D(q) = doq™" + dq”! +-+++dm. 
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Assume that e(t) is the same in Equations 60.46 and 60.47 and that it is white noise of zero mean and unit 
variance. The problem is to find the cross-covariance elements 


r(k) = Ey(t+ k)w(t) (60.48) 


for a number of arguments k. The cross-covariance function r(k) is related to the cross-spectrum oy (z) 
as (see Equations 60.7 and 60.22) 


DS yet = BODEN 
by) 2, ee SCNT (60.49) 
Introduce the two polynomials 
EF =f, n n—1 are a 
CO Ae (60.50) 
GE) = gor giz OY hss ot gn az 
through 
B(z) D(z!) _ F(z) G(z7!) 
Al) Cle) ~ Aw)" “Ce” eee 
or, equivalently, 
B(z)D(z7!) = F(z)C(z7) + zA(z)G(z71). (60.52) 


Since zA(z) and C(z~!) are coprime (i.e., these two polynomials have no common factor), Equation 60.52 
has a unique solution. Note that as a linear system of equations, Equation 60.52 has n+ m-+ 1 equations 
and the same number of unknowns. The coprimeness condition ensures that a unique solution exists. 
Equations 60.49 and 60.51 now give 


* -¢_ F(z) | Gz) 
> r(k)z* = At + G3)" (60.53) 


k=—0o 


The two terms in the right-hand side of Equation 60.53 can be identified with two parts of the sum. In 
fact, 


F(z) _, 2G(z7) = _k 
nee kz, = kz. 60.54 
Gay 2 ey = (60.54) 
k=0 k=—00 
Equating the powers of z gives 
Tyw(0) = fo, 
tyw(1) = fi — air(0), 
(60.55) 
tyw(k) = fe — pe ajr(k—j), (2<k<n), 
ryw(k) =— SL, air(k— jy (k> 2). 
Note that the last part of Equation 60.55 is nothing but a Yule-Walker type of equation. 
Similarly, 
tyw(—1) = 80> 
tyw(—k) = ge—-1 — Baa cr(—k+j), (2<k<m), (60.56) 


ryw(—k) = — Dt gr(-k+j), (k>m). 
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Example 60.4: 


Consider again a first-order ARMA process 
y(t)+ay(t —1)=e(t)+ce(t—1), Ee2(t)=1. 


In this case, the autocovariance function is sought. Hence choose z(t) = y(t) and thus A(q) = q+, 
B(q)=q+¢,C(q) = q+a,and D(q) = q +c. The identity (Equation 60.52) becomes 


(2+ 027-1 +05 (fozt+flz~! +a) +212 +a)(goz~!). 


Equating the powers of z leads to 


pairs — 2adc fie (c — a)(1 — ac) 
o= heaa2 , 1=C Jo 1a 
Hence Equation 60.55 implies that 
1+c¢2 —2ac 
r(0) = fo = 1_ a2 ! 
(c — a)(1 — ac) 


while Equation 60 56 gives 


Needless to say, these expressions for the covariance function are the same as those derived in the 
previous example. 


60.6 Continuous-Time Processes 


This section illustrates how some of the properties of discrete-time stochastic systems appear in an 
analog form for continuous-time models. However, white noise in continuous time leads to considerable 
mathematical difficulties that must be solved in a rigorous way. See [1] or [7] for more details on this 
aspect. 

The covariance function of a process y(t) is still defined as (cf, Equation 60.2) 


r(t) = Ey(t+ t)y(t), (60.57) 


assuming for simplicity that y(t) has zero mean. The spectrum will now be 


CO 
(s) =i r(t)e “dt, (60.58) 
—0O 
and the spectral density is 
CO 
(iw) =| (ted. (60.59) 
—0o 


The inverse relation to Equation 60.58 is 


ce 7 b(s)eds, (60.60) 
2ui 


where integration is along the whole imaginary axis. 
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Consider a stationary stochastic process described by a spectral density (iw) that is a rational function 
of iw. By pure analogy with the discrete-time case it is found that 


. B(iw)B(—iw) 
where the polynomials 
A(p) =p" tap" | +---+4n 
aie ; (60.62) 


B(p) = dip” | +---+ by, 


have all their roots in the left half-plane (i-e., in the stability area). Here p is an arbitrary polynomial 
argument, but can be interpreted as the differentiation operator (py(t) = y(t)). 

The effect of filtering a stationary process, say u(t), with an asymptotically stable filter, say H(p), can 
be easily phrased using the spectra. Let the filtering be described by 


y(t) = H(p)u(t). (60.63) 
Then 
by(s) = H(s)H(—s) ox (s), (60.64) 


again paralleling the discrete-time case. As a consequence, one can interpret any process with a rational 
spectral density (Equation 60.61) as having been generated by filtering as in Equation 60.63 by using 


Bip) 


HWS AG). 


(60.65) 


The signal u(t) would then have a constant spectral density, (iw) = 1. As for the discrete-time case, 
such a process is called white noise. It will have a covariance function r(t) = 8(t) and hence in particular 
an infinite variance. This indicates difficulties to treat it with mathematical rigor. 


60.7 Optimal Prediction 


Consider an ARMA process, 
A(q)y(t) = C(qe(t), Ee*(t) =’, (60.66) 


where A(q) and C(q) are of degree n, and have all their roots inside the unit circle. We seek a k-step 
predictor, that is, a function of available data y(t), y(t — 1),... that will be close to the future value 
y(t +k). In particular, we seek the predictor that is optimal in a mean square sense. The clue to finding 
this predictor is to rewrite y(t + k) in two terms. The first term is a weighted sum of future noise values, 
{e(t+ Avs As this term is uncorrelated to all available data, it cannot be reconstructed in any way. 
The second term is a weighted sum of past noise values {e(t — s)}°°). By inverting the process model, the 
second term can be written as a weighted sum of output values, {y(t — s)}°,. Hence, it can be computed 
exactly from data. 
In order to proceed, introduce the predictor identity 


z-1C(z) = A(z)F(z) + L(z), (60.67) 
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where 


F(z) =z 4 fick? 4.0.4 fh, (60.68) 
PEC Wet Oe Aaa Nop aac Sa es (60.69) 


Equation 60.67 is a special case of a Diophantine equation for polynomials. A solution is always possible. 
This is analogous to the Diophantine equation 


n=qm+r 


for integers (for given integers n and m, there exist unique integers q and r). Now, 


C(q) q*!C(q) 
yt+h)= Ata ia A@) ———*e(t+ 1) 
= eee N=F@et+y+ © ae (60.70) 
qL(q) A(q) ql(q) 
= F(q)e(t + 1) + —— Aa) C(q)’ y(t) = F(qje(t +1) + Cig)? y(t). 


This is the decomposition mentioned above. The term F(q)e(t+ 1) is a weighted sum of future noise 
values, while qL(q)/C(q)y(t) is a weighted sum of available measurements Y‘. Note that it is crucial for 
stability that C(q) has all zeros strictly inside the unit circle (but that this is not restrictive due to spectral 
factorization). As the future values of the noise are unpredictable, the mean square optimal predictor is 
given by 

qL(q) 


Si py 
yt+k\t) = C(q) 


—— y(t), (60.71) 
while the associated prediction error is 


Y(t +k) = F(gje(t + 1) 


(60.72) 
=e(t+k)t+fre(t+k—-1)+---+fere(t +1), 
and has variance 
Pr+h=V1+fp+--+f2 4). (60.73) 
As a more general case, consider prediction of y(t) in the ARMAX model 
A(q)y(t) = B(q)u(t) + C(q)e(t). (60.74) 
In this case, proceeding as in Equation 60.70, 
B(q) qL(q) Ee Bq) 
k k)+F 
yE+kh = WG Atay +k) + F(qje(t+ 1) A(@) | O@) (t) Clg) u(t) eas 
= qL(q) qB(q)F(q) 
= F(gje(t +1) + Cla) y(t) + Cig) u(t). 


We find that the prediction error is still given by Equation 60.72, while the optimal predictor is 


ql(q) qB(q)F(q) 
cg t c@ 


This result can also be used to derive a minimum output variance regulator. That is, let us seek a feedback 
control for the process (Equation 60.74) that minimizes Ey*(t). Let k = deg A — deg B denote the delay 


H(t + kt) = u(t). (60.76) 
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in the system. As (t+ k) and p(t + k|t) are independent, 
Ey?(t +k|t) = Ey?(t + k|t) + Ey’(t +k) > Ey’ (t +h), (60.77) 
with equality if and only if }(t + k|t) = 0, the regulator is 


“oat y(t). (60.78) 


B(q)F(q) 
Optimal prediction can also be carried out using a state-space formalism. It will then involve computing 


the Kalman filter, and a Riccati equation has to be solved (which corresponds to the spectral factorization). 
See Chapter 17 for a treatment of linear quadratic stochastic control using state-space techniques. 


60.8 Wiener Filters 


The steady-state linear least-mean square estimate is considered in this section. It can be computed using 
a state-space formalism (like Kalman filters and smoothers), but here a polynomial formalism for an 
input-output approach is utilized. In case time-varying or transient situations have to be handled, a 
state-space approach must be used. See Chapter 12 for a parallel treatment of Kalman filters. 

Let y(t) and s(t) be two correlated and stationary stochastic processes. Assume that y(t) is measured 
and find a causal, asymptotically stable filter G(q) such that G(q)y(t) is the optimal linear mean square 
estimator, that is, it minimizes the criterion 


V = Els(t) — G(gy(t)P. (60.79) 


This problem is best treated in the frequency domain. This implies in particular that data are assumed to 
be available since the infinite past t = —oo. Introduce the estimation error 


3(t) = s(t) — G(q)y(t). (60.80) 


The criterion V, Equation 60.79, can be rewritten as 


yee § ree aa (60.81) 
2m 4 


Next note that 
3(z) = s(z) — G(z)bys(z) — bsy(z)G(z*) + G(z)G(z") by (2). (60.82) 


Now let G(q) be the optimal filter and G;(q) any causal filter. Replace G(q) in Equation 60.79 by G(q) + 
€G,(q). As a function of ¢, V can then be written as V = Vo + eV) + &°V>. For G(q) to be the optimal 
filter it is required that V > Vo for all e, which leads to V; = 0, giving 


d. 
0= tr § [G(2)b,(2) — dylZ1G eS. (60.83) 


Zz 


It is possible to give an interpretation and alternative view of Equation 60.83. For the optimal filter, the 
estimation error, s(t), should be uncorrelated with all past measurements, {y(t — DY o- Otherwise there 
would be another linear combination of the past measurements giving smaller estimation error variance. 


Linear Stochastic Input-Output Models 60-15 


Hence, 


EX(t)y(t —j)=0, all j>0, (60.84) 


or 


Es(t)[Gi(q)y(t)] =0 for any stable and causal Gj)(q). (60.85) 


This can be rewritten as 
0 = E[s(t) — G(q)y(®) Gi (qy@)] 
dz (60.86) 


— = $loy(2) a G(z)by(z)]Gy (hH_—, 


Zz 


which is precisely (60.83). 
From Equation 60.83, one easily finds the unrealizable Wiener filter. Setting the integrand to zero gives 
G(z)by(z) = Hsy(Z), and 


G(z) = dbsy(z), '(2). (60.87) 


The filter is not realizable since it relies (except in very degenerate cases) on all future data points of y(t). 
Note, however, that when “deriving” Equation 60.87 from Equation 60.83, it was effectively required that 
Equation 60.83 holds for any G;(z). However, it is only required that Equation 60.83 holds for any causal 
and stable G,(z). Such an observation will eventually lead to the optimal realizable filter. 

To proceed, let the process y(t) have the innovations representation (remember that this is always 
possible as in Section 60.4) 


y(t) = H(q)e(t), 


; (60.88) 
Ee(t)e(s) = “81s,  H(0O)=1, 
H(q), H~'(q) asymptotically stable. 
Then y(z) = H(z)H (z~!)22. Further, introduce the causal part of an analytical function. Let 
2°. r 
Ga@)= Yo gz, (60.89) 


j=-0 


where it is required that the series converges in a strip that includes the unit circle. The causal part of 
G(z) is defined as 


[G(z)]+ = Do giz, (60.90) 
j=0 


and the anticausal part is the complementary part of the sum: 


-1 
[G@)_-= 0 gz? = G@)-[G@)+. (60.91) 


j=—00 


It is important to note that the term goz~° in Equation 60.89 appears in the causal part, [G(z)]+. Note 
that the anticausal part [G(z)]_ of a transfer function G(z) has no poles inside or on the unit circle, and 
that a filter G(z) is causal if and only if G(z) = [G(z)]+. Using the conventions (Equation 60.90) and 
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(Equation 60.91), the optimality condition (Equation 60.83) can be formulated as 


0= © $1G@HE) ~ b(t NH eYoE (eh) Z 
2mi Zz 


= 5 P (GHG) ~ [5 (2) HED 
Tl 


d 

= [by (HAY AE DEEDS. (60.92) 
Zz 

The stability requirements imply that the function H(z~!)G,(z7!) does not have any poles inside the unit 

circle. The same is true for [s,(z){H (z7)}-1X-21_, by construction. The latter function has a zero at 

z = 0. Hence, by the residue theorem, 


1 dz 
aa fo OHE LHe GENS =o. (60.93) 
2ui Zz 
The optimal condition of Equation 60.92 is therefore satisfied if 
1 
G(z) = salby HE} +H"). (60.94) 


This is the realizable Wiener filter. It is clear from its construction that it is a causal and asymptotically 
stable filter. 
The Wiener filter will be illustrated by two examples. 


Example 60.5: 


Consider the ARMA process 
A(q)y(t) = C(q)e(t), 
Ee*(t) = 27, 


Treat the prediction problem 
s(t) = y(t +k), k>0, 


In this case, 


bsy(z) = Z* by(z). 
The unrealizable filter (Equation 60.87) becomes, as before, 
Gz) = 2* oylz)by (2) =z", 


meaning that 
S(t) = y(t +k). 


Note that it is noncausal, but it is otherwise a perfect estimate since it is without error! Next, the 
realizable filter is calculated: 


aoe ok 2c) Ce") Ae") M0 [aC] A(z) 
~ 32 A(z) Alz~!) C(z~") | Cz) + C2) 
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Ww | ca |_s 0) s(t-m) 
A(q) © ide 


FIGURE 60.3 Setup for a polynomial estimation problem. 


To proceed, let A(z) and C(z) have degree n, and introduce the polynomial F(z) of degree k — 1 and 
the polynomial L(z) of degree n — 1 by the predictor identity (Equation 60.67). This gives 
G(z) = ee | A(z) zL(z) A(z) _ zL(z) 
2 A(z) 4 C(z) A(z) Cz) C2)" 


The optimal predictor therefore has the form 


5(t) = y(t + k|t) = —— yit) 


in agreement with Equation 60.71. 


Example 60.6: 


Consider the measurement of a random signal s(t) in additive noise w(t). This noise source need 
not be white, but it is assumed to be uncorrelated with the signal s(t). Model s(t) and w(t) as ARMA 
processes; see Figure 60.3. Thus, 


y(t) =s(t)+w(t), s(t)= qa") w(t) = eet 
Ev(t)v(s) =28¢5, Ee(t)e(s)=28¢5, Ee(t)v(s) = 0. 


The polynomials in the model are 


Cq=q +aq  +:::+Cn, 
Aq) =q" +a1q" '4--++4n, 

M(q) = qf +m’! +--+ mr, 
(q) 


N(q) =! +m’! +-+++n¢. 


Assume that all four polynomials have all their roots inside the unit circle. The problem to be treated 
is to estimate s(t — m) from {y(t — /) 0! where m is an integer. (By definition, m = 0 gives filtering, 
m > Osmoothing, and m < 0 prediction.) 


To solve the estimation problem, first perform a spectral factorization of the output spectrum: 
Ole: ) 2 M@M(z*) re B(z)B(z') 
VA(z)A(z-!)  ® N(z)N(z7!) A(z) N(z) A(z) N(z78)’ 


requiring that B(z) is a monic polynomial (i.e., it has a leading coefficient equal to one) of degree n +1, 
and that it has all zeros inside the unit circle. The polynomial B(z) is therefore uniquely given by the 


y(z) =k 
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identity 
02 B(z)B(z~!) = 02.C(z)C(z7!)N(z)N(z7!) + A(Z)A(z!)M(z)M(z7}). 
Hence, ; 
B(z) 2 2 ama C(z)C(z~") 
H(z)= A@NG’ M=E,  Vsy(z) =z \VA@AG=) 


According to Equation 60.94, the optimal filter becomes 


eae a, ae C(z)C(z7!) A] A(z)N(z) 
n2 ” A(z)A(z7!) Biz") J, BY) 
MP em C@Ce)N(z™")] AZ) N(z) 
2 A(z)B(z~!) , Be) 


The causal part [ ]4 can be found by solving the Diophantine equation 
Z ™O@eZ NE Sz Ber Ri) 4 2PM A@Le-); 
where the unknown polynomials have degrees 
deg R= n— min(0,—m), 
deg L=n+r—1+max(0,—m). 


« 


Note that the “—1” that appears in deg L has no direct correspondence in deg R. The reason is that the 
direct term goz~° in Equation 60.89 is associated with the causal part of G(z). 
The optimal filter is readily found: 


n2 (ee Se] eS 
+ 


2 
€ 


GES <5 A@Be—) A@Bz—) B) 


n2 | zminO—m) Riz) | A(z)N(z) 2 zmin®@-™) R(z)N(z) 
ee A(z) Biz) 2 B(z) 
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61.1 Introduction 


Dynamic programming is a recursive method for obtaining the optimal control as a function of the state 
in multistage systems. The procedure first determines the optimal control when there is only one stage 
left in the life of the system. Then it determines the optimal control where there are two stages left, etc. 
The recursion proceeds backward in time. 

This procedure can be generalized to continuous-time systems, stochastic systems, and infinite horizon 
control. The cost criterion can be a total cost over several stages, a discounted sum of costs, or the average 
cost over an infinite horizon. 

A simple example illustrates the main idea. 


61.1.1 Example: The Shortest Path Problem 


A bicyclist wishes to determine the shortest path from Bombay to the Indian East Coast. The journey can 
end at any one of the cities N3, C3, or S3. The journey is to be done in 3 stages. 

Stage zero is the starting stage. The bicyclist is in Bombay, labelled Cp in Figure 61.1. Three stages of 
travel remain. The bicyclist has to decide whether to go north, center, or south. If the bicyclist goes north, 
she reaches the city Nj, after travelling 600 kms. If the bicyclist goes to the center, she reaches Cj, after 
travelling 450 kms. If she goes south, she reaches S) after travelling 500 kms. 

At stage one, she will therefore be in one of the cities Nj, Cy, or S;. From wherever she is, she has to 
decide which city from among Np, C2, or Sz she will travel to next. The distances between cities Nj, Cj, 
and S; and N2, C2, Sz are shown in Figure 61.1. At stage two, she will be in one of the cities Nz, C2, or 
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FIGURE 61.1 A shortest path problem. 


Sy, and she will then have to decide which of the cities N3, C3, or S3 to travel to. The journey ends in 
stage three with the bicyclist in one of the cities N3, C3 or S3. 


61.1.2 The Dynamic Programming Method 


We first determine the optimal decision when there is only one stage remaining. 

If the bicyclist is in Nz, she has three choices—north, center, or south, leading respectively to N3, C3 or 
S3. The corresponding distances are 450, 500 and 650 kms. The best choice is to go north, and the shortest 
distance from N> to the East Coast is 450 kms. 

Similarly, if the bicyclist is in C2, the best choice is to go center, and the shortest distance from C2 to the 
East Coast is 550 kms. From Sp, the best choice is to go south, and the shortest distance to the East Coast 
is 400 kms. We summarize the optimal decisions and the optimal costs, when only one stage remains, in 
Figure 61.2. 

Now we determine the optimal decision when two stages remain. 
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FIGURE 61.2 Optimal solution when one stage remains. 
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FIGURE 61.3 Making a decision in city Nj. 


Suppose the bicyclist is in N. If she goes north, she will reach Np after cycling 600 kms. Moreover from 
Np, she will have to travel a further 450 kms to reach the East Coast, as seen from Figure 61.2. From Nj, 
if she goes north, she will therefore have to travel a total of 1050 kms to reach the East Coast. 

If instead she goes to the center from Nj, then she will have to travel 650 kms to reach C2, and from 
C> she will have to travel a further 550 kms to reach the East Coast. Thus, she will have to travel a total of 
1200 kms to reach the East Coast. 

The only remaining choice from Nj is to go south. Then she will travel 750 kms to reach S), and from 
there she has a further 400 kms to reach the East Coast. Thus, she will have to travel 1150 kms to reach 
the East Coast. 

The consequences of each of these three choices are shown in Figure 61.3. Thus the optimal decision 
from Nj is to go north and travel to Nz. Moreover, the shortest distance from Nj, to the East Coast is 
1050 kms. 

Similarly, we determine the optimal paths from C, and Sj, also, as well as the shortest distances from 
them to the East Coast. The optimal solution, when two stages remain, is shown in Figure 61.4. 

Now we determine the optimal decision when there are three stages remaining. 
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FIGURE 61.4 Optimal solution when two stages remain. 
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The bicyclist is in Co. If she goes north, she travels 600 kms to reach Nj, and then a further 1050 kms is 
the minimum needed to reach the East Coast. Thus the total distance she will have to travel is 1650 kms. 
Similarly, if she goes to the center, she will travel a total distance of 1450 kms. On the other hand, if she 
goes south, she will travel a total distance of 1400 kms to reach the East Coast. Thus the best choice from 
Co is to travel south to S), and the shortest distance from Cp to the East Coast is 1400 kms. This final 
result is shown in Figure 61.5. 


61.1.3 Observations on the Dynamic Programming Method 


We observe a number of important properties. 


1. In order to solve the shortest path from just one city Co to the East Coast, we actually solved the 
shortest path from all of the cities Cp, Ni, Ci, S1, N2, C2, Sz to the East Coast. 

2. Hence, dynamic programming can be computationally intractable if there are many stages and 
many possible decisions at each stage. 

3. Dynamic programming determines the optimal decision as a function of the state and the number 
of stages remaining. Thus, it gives an optimal closed-loop control policy. 

4. Dynamic programming proceeds backward in time. After determining the optimal solution when 
there are s stages remaining, it determines the optimal solution when there are (s+ 1) stages 
remaining. 

5. Fundamental use is made of the following relationship in obtaining the backward recursion: 


Optimal distance from state x to the end 
Distance travelled on current stage when decision dj is made + optimal 
= min} distance from the state x’ to the end, where x’ is the state to which decision 
‘ d; takes you from state x 


Hence, if one passes through state x’ from x, an optimal continuation is to take the shortest path 
from state x’ to the end. This can be summarized by saying “segments of optimal paths are optimal 
in themselves.” This is called the Principle of Optimality. 
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FIGURE 61.5 Optimal solution when three stages remain. 
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61.2 Deterministic Systems with a Finite Horizon 


We can generalize the solution method to deterministic, time-varying control systems. Consider a system 
whose state at time t is denoted by x(t) € ¥. If a control input u(t) € U/ is chosen at time f, then the next 
state x(t-+ 1) is determined as, 


x(t +1) = f(x(#), u(t), t). 


The system starts in some state x(fo) = xo at time fp. For simplicity, suppose that there are only a finite 
number of possible control actions, i-e., U/ is finite. 

The cost incurred by the choice of control input u(t) at time ft is c(x(t), u(t)). We have a finite time 
horizon T. The goal is to determine the controls to minimize the cost function, 


T-1 


Y- c(x(t), u(t), t) + d(x(T)). (61.1) 


t=to 


Theorem 61.1: Deterministic System, Finite horizon, Finite Control Set 


Define 
V(x, T):= d(x), and (61.2) 
V(x,t—-1):= _ {c(x, u,t—1)+ V(f(x,u,t — 1), t)} : (61.3) 


Let u*(x, t — 1) be a value of u which minimizes the right-hand side (RHS) above. Then, 


1. V(x, t) = min, ye} c[x(n), u(n), n] + d[x(T)], when x(t) = x, that is, it is the optimal cost-to-go 
from state x at time t to the end. 
2. u*(x,t) is an optimal control when one is in state x at time t. 


Proof 61.1. Define V(x, t) as the optimal cost-to-go from state x at time t to the end. V(x, T) = d(x), 
because there is no decision to make when one is at the end. Now we obtain the backward recursion of 
dynamic programming. Suppose we are in state x at time tf — 1. If we apply control input u, we will incur 
an immediate cost c(x, u, tf — 1), and move to the state f(x, u, t — 1), from which the optimal cost-to-go 
is V(f (x, u,t — 1), t). Because some action u has to be taken at time ¢ — 1, 


V(x,t-1)< min| es u,t—1)+ V(f(x,u,t — v»o}. 


Moreover, if V(x, t — 1) was strictly smaller than the RHS above, then one would obtain a contradiction 
to the minimality of V(-, t). Hence equality holds, and Equation 61.3 is satisfied. 

By evaluating the cost of the control policy given by u*(x, t), one can verify that it gives the minimal 
cost V(x0, to). 


We have proven the Principle of Optimality. We have also proven that, for the cost criterion Equa- 
tion 61.1, there is an optimal control policy that takes actions based only on the value of current state and 
current time and does not depend on past states or actions. Such a control policy is called a Markovian 
policy. 
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61.2.1 Infinite Control Set U/ 


When the control set U/ is infinite, the RHS in Equation 61.3 may not have a minimizing u. Thus the 
recursion requires an “inf” in place of the “min.” There may be no optimal control law, only nearly 
optimal ones. 


61.2.2 Continuous-Time Systems 


Suppose one has a continuous-time system 
x(t) = f (x(t), u(t), t), (61.4) 


and the cost criterion is 
E 


ats(T))-+ f elelt)u(t)t dt. 
to 

Several technical issues arise. The differential Equation 61.4 may not have a solution for a given u(t), or 

it may have a solution but only until some finite time. Ignoring such problems, we examine what type 

of equations replace the discrete-time recursions. Clearly, V(x, T) = d(x). Considering a control input u 

which is held constant over an interval [t, f + A), and approximating the resulting state (all possible only 

under more assumptions), 


V(x, t)= iy {c(x, u,thA+ V(x+f(x,u,t)A,t+ A)+ o(A)} P 
ue 
ov OV 
= inf {ets u,t)A+ V(x, t)+ a t)f (x, u, t)A + a thAa+ o(a)} : 
Thus, one looks for the partial differential equation 
ov OV 
ae t)+ int| a” t)f (x, u, t) + c(x, u, of =0, (61.5) 


to be satisfied by the optimal cost-to-go. This is called the Hamilton-Jacobi-Bellman equation. To prove 
that the optimal cost-to-go satisfies such an equation requires many technical assumptions. 

However, if a smooth V exists that satisfies such an equation, then it is a lower bound on the cost, at 
least for Markovian control laws. To see this, consider any control law u(x, f), and let x(t) be the trajectory 
resulting from an initial condition x(to) = xo at time to. Then, 


cl) te oe, t)f (x(t), u(x(t), t), t) + c(x(t), u(x(£), t) = 0. (61.6) 


Hence FLV(x(t), t)] + c(x(t), u(x(t), t), t) = 0, and so, 


T 
V(x(ty), to) < V(e(T), T) + / c(x(t), u(t), t) dt 


to 
T 


=ax(r + [ c(x(t), u(t), t) dt. 


to 


Moreover, if u*(x, t) is a control policy which attains the “inf” in Equation 61.5, then it attains equality 
above and is, hence, optimal. 

In many problems, the optimal cost-to-go V(x, f) may not be sufficiently smooth, and one resorts to 
the Pontryagin Minimum Principle rather than the HJB Equation 61.5. 
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61.3 Stochastic Systems 


Consider the simplest case where time is discrete, and both the state-space V and the control set U/ are 
finite. Suppose that the system is time invariant and evolves probabilistically as a controlled Markov chain, 
Le., 


Prob[x(t + 1) = jlx(t) =i, u(t) =u] = pi(u). (61.7) 


The elements of the matrix P(u) = [pj(u)] are the transition probabilities. 
Given a starting state x(to) = xo at time fo, one wishes to minimize the expected cost 


T-1 
E bs c(x(t), u(t), t) + aor 


t=to 


A nonanticipative control policy y is a mapping from past information to control inputs, ie, y= 
(Vtg> Viot1> +++» ¥T—1) where yt : (x(to), u(to), x(to + 1), u(to + 1),...,x(f)) > u(t). Define the condition- 
ally expected cost-to-go when policy y is applied, as 


T-1 
vy =E be c(x(n), u(n), n) + d(x(T))||x(to), u(to), .. « 0) : 


n=t 


Analogously to the deterministic case, set Equation 61.2, and recursively define 


V(x,t—1)= min ot u,t—1)+ Dd PwVG, o}. (61.8) 
J 


Now let us compare vy with V(x(t), t) when policy y is used. Clearly VY = V(x(T), T) = d(x(T)) as. 
Now suppose, by induction, that V;’ > V(x(t), f) a.s. Then 


T 
Vi = c(x(t—1),u(t—1),t -—1) +E ps c[x(n), u(n), n) + d(x(T)]|x(to), u(to), ..., x(t — »| 


n=t 


T 
=c(x(t—1),u(t—1),t-1)+E {2 bs c[x(n), u(n), n) + d(x(T)]|x(to), u(to), .- « 0) 


n=t 


x(to), u(to),...,x(t — »| 


= e(x(t — 1), u(t — 1),t —1) + E[V}|x(to), u(to), ....x(t — 1)] (61.9) 
> c(x(t — 1), u(t — 1), t — 1) + E[V(x(0), £)|x(to), u(to), ...,x(t — 1)] 


= c(x(t—1),u(t—1),t- 1) + D> pee—njlult -— DIVG. 1) 
j 


> min c(x(t—1),u,t-1)+ Do Pxe-ryVU, t) 
j 


= V(x(t), ft) as. 


Thus V(x,t) is a lower bound on the expected cost of any non anticipative control policy. Suppose, 
moreover, that u* (x, t) attains the minimum on the RHS of Equation 61.8 above. Consider the Markovian 
control policy y* = (yj, V;,>---»Vip_,) with 


Vi Lx(to), u(fo), .- > x(2)] = u*[x(0), f]. (61.10) 


For y* it is easy to verify that the inequalities in Equation 61.9 are equalities, and so it is optimal. 


61-8 Control System Advanced Methods 


Theorem 61.2: Stochastic Finite State, Finite Control System, Finite Horizon 


Recursively define V(x, t) from Equations 61.2 and 61.8. Let u*(x,t — 1) attain the minimum on the RHS 
of Equation 61.8, and consider the Markovian control policy y* defined in Equation 61.10. Then, 


i. V(x, to) is the optimal cost. 
ii. The Markovian control policy y* is optimal over the class of all nonanticipative control policies. 


61.3.1 Countably Infinite State and Control Spaces 


The result (i) can be extended to countably infinite state and control policies by replacing the “min” above 
with an “inf”. However, the “inf” need not be attained, and an optimal control policy may not exist. 

If one considers uncountably infinite state and control spaces, then further highly technical issues 
arise. A policy needs to be a measurable map. Moreover, V(x, t) will also need to be a measurable function 
because one must take its expected value. One must impose appropriate conditions to insure that the 
“inf” over an uncountable set still gives an appropriately measurable function, and further that one can 
synthesize a minimizing u(x, t) that is measurable. 


61.3.2 Stochastic Differential Systems 
Consider a system described by a stochastic differential equation, 
dx(t) = f (x(t), u(t), t) dt + dwi(t), 


where w is a standard Brownian motion, with a cost criterion 
T 


E{d[x(T)]+ / cLx(t), u(t), tI}. 


to 
If V(x, t) denotes the optimal cost-to-go from a starting state x when there are t time units remaining, 
one expects from Ito’s differentiation rule that it satisfies the stochastic version of the Hamilton-Jacobi- 
Bellman equation: 
AV(x,t) 10V 


Ot 2 Ox? 


The existence of a solution to such partial differential equations is studied using the notion of viscosity 


. OV 
(x,t) 4 int | BE (x, t)f (x, u, t) + c(x, u, o} =0. 
solutions. 


61.4 Infinite Horizon Stochastic Systems 


We will consider finite state, finite control, controlled Markov chains, as in Equation 61.7. We study the 
infinite horizon case. 


61.4.1 The Discounted Cost Problem 


Consider an infinite horizon total discounted cost of the form 


+00 
EY Bie(x(t), w(t)), (61.11) 
t=0 
where 0 < B < 1 is a discount factor. The discounting guarantees that the summation is finite. We are 
assuming that the one-stage cost function c does not change with time. Let W(x, oo) denote the optimal 
value of this cost, starting in state x. The “oo” denotes that there is an infinity of stages remaining. 
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Let W(x, N) denote the optimal value of the finite horizon cost E ye B'c[x(t), u(t)], when starting 
in state x at time 0. Note that the index N refers to the number of remaining stages. From the finite 
horizon case, we see that 


Ws) = min ols u) + Y | PaiwBW GN — vf. (61.12) 
J 


Note the presence of the factor B multiplying W(j, N — 1) above. 
Denote by R* the class of all real valued functions of the state, ie., V € R® is a function of the form 
V: ¥ > R. Define an operator T : R* > R* by its action on V as 


TV(x):= mines u) +8 Cewvo forallxe X. (61.13) 
J 


Using the operator T, one can rewrite Equation 61.12 as W(-.,N) = TW(-,N — 1). Also, W(-,0) =O, 
where O is the identically zero function, i.e., O(x) = 0 for all x. Note that T is a monotone operator, i.e., 
if V(x) < V(x) for all x, then TV(x) < TV(x) for all x. Let us suppose now that ¢ > c(x, u) > 0 for all x, 
u. (This can be achieved by simply adding a large enough constant to each original one-step cost.) Due 
to this assumption, TO > 0. Hence by monotonicity, TMO>TA-VOS..-> TO> 0. Thus TN) O(x) 


converges, for every x, to a finite number (since TO< rR): Let 
W(x) := lim T)O(x). 
N->0o 


Now W(x, co) > W(x, N) for all N, because c(x, u) > 0. Hence W(x, 00) > limy_so09 W(x, N) = W(x). 
zpN 
Moreover W(x, 00) < W(x,N)+ oo because one can employ an optimal policy for an N-stage problem. 


Thus W(x, co) < limy-5o [We N)+ | = W(x). Hence W(x) = W(x, co), and is, therefore, the 
optimal infinite horizon cost. 


Hence, we would like to characterize W(x). Since T is continuous, TW(x) = limy-5o0 TNt+!O(x) = 
W(x). Hence W is a fixed point of T, ie., 


W(x) = in| +B Drsww}. (61.14) 
J 


Simple calculations show that T is a contraction with respect to the || - ||oo norm, ie., max, |TV (x) — 
TV(x)| < 8 max, |V(x) — V(x)|. Thus T has a unique fixed point. Since T is a contraction, we also know 
that limy_.9, T®) V = W for any V. 

Suppose now that u*(x) attains the minimum in Equation 61.14 above. Consider the policy y* which 
always chooses control u*(x) whenever the state is x. Such a control policy is called stationary. If one 
applies the stationary control policy y*, then the expected cost over N days, starting in state x, is TY O(x). 
Thus the infinite horizon cost of y* is limy_.o, TYO = W. 


Theorem 61.3: Stochastic Finite State, Finite Control System, Discounted Cost 
Criterion 


Let T : R* — R* denote the operator (Equation 61.13). 


1. T is a contraction. 
2. Let W =limy-.o0 TV for any V. Then W is the unique solution of Equation 61.14. 
3. W(x) is the optimal cost when starting in state x. 
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4. Let y*(x) be the value of u which attains the minimum on the RHS in Equation 61.14. Then y* is a 
stationary control policy which is optimal in the class of all nonanticipative policies. 


The procedure for determining W(x) as limy—.o9 TO is called value iteration. Another procedure 
which determines the optimal policy in a finite number of steps is policy iteration. 


61.4.1.1 Policy Iteration Procedure 
For a stationary policy y, define Ty : R* > R* by, T, V(x) = c(x, y(x)) +B dj Pxi(Vx)) VQ). 


1. Let yo be any stationary policy. 
2. Solve the linear system of equations Ty, Wy, = W,, to determine its cost W,,. 


3. If TW,, A Wy,, then let yj (x) be the value of u which attains the minimum in min, <7, {ets u)+ 


DOL) 


4, Then y; isa strict improvement of yo (since Wy, = limy-+o0 ja Wyo i Wy,). 
5. By repeating this procedure, one obtains a sequence of strictly improving policies, that must 
terminate in a finite number of steps, because the total number of stationary policies is finite. 


61.4.2 The Average Cost Problem 


We consider the average cost per unit time over an infinite horizon, 


, Na 
lim sup ae 2S c[x(t), u(t)]. 


N->oo t—0 


Then the dynamic programming equation needs to be modified slightly. 


Theorem 61.4: 


Suppose a constant J* exists, and a function V : X — R exists so that 


r+ V(x) = mines u)+ Yeswvia. (61.15) 
J 


1. Then J* is the optimal value of the average cost criterion, starting from any state x. 
2. Let y*(x) be a value of u which minimizes the RHS in Equation 61.15 above. Then y* is a stationary 
policy which is optimal in the class of all nonanticipative policies. 


Proof 61.2. Consider any nonanticipative policy. Then 
T+ V(x(t)) < c(x(t), u(t) + D> pacy(u()) VY). 
j 


Noting that E ye Pxyj (u(t) Vf) = E{V (x(t + 1))}, we see that 
E(c(x(t), u(t)) > J* + E(V (x(t) — E{V(x(t + 1))}. 


Hence fess 
eae aE dX c(x(t), u(t)) > J* + lim sup V(x(0)) — AS) 7 


ie 
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Thus J* is a lower bound on the average cost, starting from any state x. Moreover, if y* is the policy under 
consideration, then equality holds above, and so it is indeed optimal. 


The question, still a topic of active research, is when does a solution [J*, V(-)] exist for Equation 61.15? 
Let us consider the following simplifying assumption. For every stationary policy y, the Markov chain 
Py =[pi(y(i))] is irreducible. By this is meant that there exists a unique steady state distribution my, 
satisfying tyPy= ny, my(i)>0, 5°; ty(i) = 1, which is strictly positive, i.e., ny(i) > 0 for all i. Then 
the average cost J, starting from any state is a constant and satisfies, Jy = }°; my(i)cli, y(i)] = mycy (where 
cy :=[cy(1),...,¢,(1x)]). Hence ife=(1,..., 1), Ty(Jye — cy) = 0. Hence (Jye — cy) is orthogonal to 
the null space of (Py —I )", and is, therefore, in the range space of (Py — I). Thus a Vy exists so that 
Jye — ¢y = (Py —I)Vy, which simply means that 


Ty + Vy(x) = efx, V+ D> pail) 1Vy(i)- (61.16) 
j 


Note that V,(-) is not unique because V,(-) + a is also a solution for any a. Let us therefore fix V(x) = 0 
for some X. One can obtain a policy iteration algorithm, as well as prove the existence of J* and V(-), as 
shown below. 

61.4.2.1 Policy Iteration Procedure 


1. Let yo be any stationary policy. 
2. Solve Equation 61.16, with y replaced by yo, to obtain (Jy,, Vy,). If Jy), Vyp) does not satisfy 


Equation 61.15, then let y; (x) attain the minimum in min,¢z, | c(x, u) + i Pxj (U4) Vy (x) | 
Then yj is a strict improvement of yo. (This follows because my, (i) > 0 for all i, and so 


Jy 0 Ty, Vyo = Ty, Tye + Vy) > Ty Cy + Ty, Py, Vyo 
= Jy, + My, Vyo> and so Jy > Jy ). 


Because the policy space is finite, this procedure terminates in a finite number of steps. At termination, 
Equation 61.15 is satisfied. 


61.4.3 Connections of Average Cost Problem with Discounted Cost Problems 
and Recurrence Conditions 


The average cost problem can be regarded as a limit of discounted cost problems when B /7 1. We illustrate 
this for systems with countable state space and finite control set. 


Theorem 61.5: Connection between Discounted and Average Cost 


Let We(x) denote the optimal discounted cost E as 8 c(x(t), u(t)) when starting in the state x. Suppose 
that |We(x) — We(x’)| < M all x, x’, and allB € (1 —€«, 1), for some € > 0. For an arbitrary state x € X, let 
Bn 7 1 be a sub-sequence so that the following limits exist: 

lim (1—Bn)We,(x) =:J", and lim (We, (x) — Wg, ()) =: V(x). 

n— oo n— oo 
Then, 


1. y* 4+ V(x) = minycy {ets u) + dj PxilA) via}. 
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2. If a stationary policy y* is optimal for a sequence of discount factors By with By 7 1, then y* is 
optimal for the average cost problem. 


Proof 61.3. The dynamic programming Equation 61.14 for the discounted cost problem can be rewritten 
as, 


(1 —B)We(x) = mines u)+B Do Pal Wels) - wpcor. 
J 
Taking limits along B,, 7 1 yields the results. 


The existence of (J*, V(-)) satisfying the average cost dynamic programming Equation 61.15 is guar- 
anteed under certain uniform recurrence conditions on the controlled Markov chain. 


Theorem 61.6: Uniformly Bounded Mean First Passage Times 


Let t denote the first time after time 1 that the system enters some fixed state x, i.e., 
t= min >1: x(t) =I, 


Suppose that the mean first passage times are uniformly bounded, i.e., 
E(t | x(0) = x and yis used) < M < +00, 


or all states x and all stationary policies y. Then a solution (J*, V(-)) exists to Equation 61.15. 
YP q 


Proof 61.4. Under a stationary policy yg which is optimal for the discount factor B, 


t—1 
We(x) =E bs Bi c(x(t), u(t)) + B* Wa(%) | x(0) = ; 


t=0 
< CE[t | x(0) = x] + We(X) < CM + We(3). 


Moreover, by Jensen’s inequality, 
Wa(x) = E[B* We(X) | x(0) = x] 
= W(X)EIB™ | x(0) = x] > Wa(@)B™. 


Hence, —cM < oe < We (x)(BM — 1) < We(x) — We (x) < cM, and the result follows from the pre- 
ceding Theorem. 
61.4.4 Total Undiscounted Cost Criterion 
Consider a total infinite horizon cost criterion of the form 
+00 
E) clx(t), u(t)]. 
t=0 


In order for the infinite summation to exist, one often assumes that either 


c(x,u)>0 forallx,u (the positive cost case), or 


c(x,u) <0 forallx,u (the negative cost case). 


These two cases are rather different. In both cases, one exploits the monotonicity of the operator T. 
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Further Reading 


For discounted dynamic programming, the classic reference is Blackwell [1]. For the positive cost case, 
we refer the reader to Strauch [2]. For the negative cost case, we refer the reader to Blackwell [3,4] and 
Ornstein [5]. For the average cost case, we refer the reader to Blackwell [6] for early fundamental work, 
and to Hernandez-Lerma and Lasserre [7] and the references contained there for recent developments. 
For a study of the measurability issues which arise when considering uncountable sets, we refer the reader 
to Bertsekas and Shreve [8]. For continuous time stochastic control of diffusions we refer the reader to 
Lions [9], and to Crandall, Ishii and Lions [10] for a guide to viscosity solutions. 

For the several ways in which dynamic programming can be employed, see Bellman [11]. 

Some recent textbooks which cover dynamic programming are Bertsekas [12], Kumar and Varaiya [13], 
and Ross [14]. 
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62.1 Introduction to Approximate Dynamic Programming 


62.1.1 Approximate Dynamic Programming in Relation 
to Modern Control Approaches 


The regular approach to control system design is a two-step procedure as follows: 


1. Use modeling and/or identification techniques, of the sort presented in this handbook, to deter- 
mine a most accurate mathematical description of the system. 
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2. Employ a controller design method to determine, based on the identified model, a control strategy 
such that the prescribed performances are obtained. 


This approach at deriving control strategies is successful under the assumption that the system 
dynamics can be modeled and are not changing over time. Optimal controllers fall in this category as 
they are generally determined considering a fixed given model of the system. The goal of an optimal 
control strategy is the minimization of a cost index, which reflects the amount of energy used for control 
purposes and the distance between the present and desired performance of the controlled system. Though 
they have good robustness properties relative to possible changes in the system dynamics, optimal con- 
trollers are neither adaptive nor are they determined considering possible unmodeled dynamics. From 
this perspective one can generally say that an optimal controller is just as close to optimality as the model 
of the system, used during the design phase, is close to the real plant to be controlled. 

The class of adaptive controllers has been developed to confront the variations that occur in the system 
dynamics. In this case adaptation algorithms modify the structure and/or parametric description of the 
controller such that the performances prescribed for the control system are closely satisfied in the presence 
of variations in the system dynamics. Regular adaptive control methods are not optimal in the sense of 
minimizing a cost functional of the sort considered by optimal control. 

To confront optimality requirements in the presence of unknown or uncertain system dynamics a 
new class of control strategies has been introduced. approximate dynamic programming (ADP), [24], 
also addressed as adaptive dynamic programming, [13], or neuro dynamic programming, [7], is a hybrid 
approach to online adaptive optimal control. It combines elements of adaptive control with modeling and 
identification techniques for the purpose of obtaining, in an online fashion, optimal control strategies, 
with respect to a measurable cost index, for general systems. 

The objective of this chapter is twofold: 


¢ To motivate the ADP approach to controller design in the context of other control approaches, and 
introduce the main ingredients of ADP algorithms. 

¢ To overview the main results and provide practical ideas for the implementation of ADP algorithms 
in a comparative approach relative to known optimal control results. 


62.1.2 Reinforcement Learning 


ADP methods have their roots in the Reinforcement Learning (RL) approach to optimization. RL refers 
to the capability of an active agent, able to interact with its environment, to modify its behavior (i.e., learn 
and adapt), based on measurable reward or punishment stimuli form the environment. Such stimuli 
reflect the distance between the performances associated with the present behavior strategy, relative to a 
desired performance index. This approach to learning is constructed on the idea that successful actions 
should be remembered, based on reinforcement information, such that they are more likely to be used a 
second time. 

These methods were introduced in the computational intelligence community with the purpose of 
providing means of improvement for the behavior of an agent, which acts in an unknown and hostile 
environment. The idea originates from experimental animal learning where it has been observed that the 
dopamine neurotransmitter in the basal ganglia acts as a reinforcement signal, which favors learning at 
the level of the neuron, while encoding information on the difference between the actual and expected 
result of a performed action. 

ADP algorithms were mostly developed and successfully implemented in the framework of Markov 
decision processes (MDP), which are of interest for the computational intelligence community. In the 
MDP formulation the spaces of states of the environment and actions of the agent are discrete and 
finite. Also, the transitions between different states, conditioned by a choice of an action, are performed 
according to a probability distribution at discrete moments in time. In control engineering the interest 
shifts toward developing controllers for man-engineered systems, which evolve in continuous state and 
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action spaces in a deterministic fashion. Such systems will be the object of the ADP methods presented 
in this chapter. 

In the remaining part of this section we will present the main components of ADP methods in a 
comparative (side-by-side) treatise for discrete-time and continuous-time systems. Discussions that refer 
to discrete-time or continuous-time formulations are indicated by “DT” and “CT” at the beginning of 
the paragraphs. Sections 62.2 and 62.3 will outline several ADP methods while giving practical imple- 
mentation ideas. In Section 62.2 of this chapter we will focus our attention on ADP for discrete-time 
systems, as the implementation tools for discrete-time state-feedback controllers are more at hand in 
control engineering practice. We will then shift the focus, in Section 62.3, on the ADP methods devel- 
oped in a continuous-time framework. Further reading suggestions are included, after the bibliographical 
references, for those interested in this new and fast-growing research area. 


62.1.3 System/Environment and Control/Behavior Policy 


Most ADP algorithms were developed while considering affine in the inputs systems, which unfold their 
dynamics in discrete-time. Such systems are described by the set of state-space difference equations 


Xk =f (xk) + g(xK) UK (62.1) 


where k denotes the discrete-time index, x, € R” and u, € R”, and f(.), g(.) are nonlinear functions. 
For systems that have a continuous-time evolution of the state vector, the description is given in terms 
of a set of ordinary differential equations 


x=f(x)+g(x)u (62.2) 


where f(.), g(.) are nonlinear functions, x € R” and u € R”™. 
The goal of the ADP methods will be one of calculating optimal state-feedback controllers that map 
the state space in the action space, h: R” — R", and have the form 


un = h(x) (62.3) 


and respectively 
u = h(x). (62.4) 


A state-feedback control function will be addressed as a control policy. Given the control policy in 
Equation 62.3, the solution of the differential equation 62.1, considering initial conditions xo at time 
k =0, will be denoted by OH = (k; x, h(.)). Similarly, the solution of Equation 62.2 with control policy 
(Equation 62.4) will be ¢"(t) = 9(t; xo, h(.)). 

In order to have a well-posed problem, that is, a optimal control problem which admits a stabilizing 
control solution, we assume the system is stabilizable on some set Q € R” that includes the equilibrium 
origin. Thus, we assume that there exists a control policy of the form (Equation 62.3 or 62.4), such that 
the closed-loop system x,41 = f (xx) + g(x,)A(xx) or x = f(x) + g(x) h(x) for the continuous-time case, is 
asymptotically stable on Q. 

Generally, for a given system, a state-feedback control policy can be determined using various methods 
of design presented in this handbook. In line with the RL approach to control, where the control policy 
is learned in real time based on stimuli from the environment, the goal of the ADP methods of design 
presented in this chapter is to determine optimal state-feedback policies based on online measured 
performance information. 


Model Free Feature of RL Algorithms Desired for ADP Methods 
In RL the agent learns the control policy, based on reinforcement stimuli, and makes use of this mapping to 
determine and perform the control actions, relative to each location in the state space of the environment, 


62-4 Control System Advanced Methods 


such that the highest amount of reward is achieved over time. Everything outside the agent is considered 
to be part of the unknown environment. For this reason in standard RL algorithms an explicit model of 
the system is never required. This is an attractive feature of RL algorithms and is desired to be maintained 
by the ADP algorithms developed in the control systems community. By considering the system that 
has to be controlled as being part of the unknown environment, such ADP algorithms would be model- 
independent approaches to optimal control solutions. 

In general the ADP methods will be developed considering a model of the system. This is desired for 
theoretical analysis in order to provide proofs of stability and performance. Nonetheless, as we shall see 
in the next sections, for the implementation of the ADP algorithms this requirement can be removed in 
some cases, the result being a model-free or partially model-free adaptive controller. 


62.1.4 Control System Performance and Bellman’s Optimality Principle 
62.1.4.1 Cost/Value Function 


The notion of optimal performance of a certain control system is captured by defining a performance 
measure of the sort used for optimal control purposes. 


DT 


In a discrete-time framework the cost function is defined as 


Valen) = Do yh r(x, us) (62.5) 
i=k 


where uz = h(x,) is a feedback control policy, r(x;, uz) is a function which quantizes the one-step rein- 
forcement received from the environment and y, which satisfies 0 < y < 1, isa discount factor. This cost 
function is also known as the cost-to-go and is a sum of discounted future reinforcement values r(x,, Ux) 
from the current time k into the infinite horizon future. The discount factor reflects the fact that reinforce- 
ment values, further into the future, are less important than immediate reinforcements. For simplicity of 
the derivation we will consider in the following that the discount factor is y = 1. 

The reinforcement function r(x,, ux) is also known as the utility, and is a measure of the one-step cost 
of the control policy. A standard form for the reinforcement function is the quadratic energy function 
(xk, UK) = cra Qxzp + uj Rug, or the more general form 


r(xp, Up) = Q(x) + uf Ru. (62.6) 


In Equation 62.6 the matrix valued function Q(x) is such that Vx 4 0, Q(x) > Oandx = 0 > Q(x) =0, 
and the matrix R is positive-definite such that the cost function (Equation 62.5) is well defined. 


CT 


In a continuous-time framework, the infinite horizon integral cost associated with the control policy 
u = h(x) is 


V"(x(t)) = i r(x(t), u(t))dt (62.7) 
t 


where x(t) denotes the solution of Equation 62.2 for initial condition x(t) and input {u(t) = h(x(t)); t > 
t}. The instantaneous reinforcement function takes the form r(x, u) = Q(x)+ u! Ru, with conditions 
Q(x) >0,R>Oandx =0> Q(x) =0. 

Admissible control policies are those maps h: R" > R™ that guarantee stability properties for the 
closed-loop system while resulting in finite value of the cost indices, Vj(xx), or respectively V"(x(t)), 
which reflect the performance of the control system. We denote with W({2) the set of all admissible 
control policies defined on a set 2 C R”. 
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We have to note here that in the cases in which the reinforcement function is interpreted as a reward 
then the cost function is called value function and the objective is to maximize it. Though in this chapter 
we will only discuss minimization problems, the extension to maximization problems is straightforward. 
However, since the ADP methods originated in the computational intelligence community where maxi- 
mization problems are generally solved, we choose to use the term of value function also in reference to the 
cost function, which is to be minimized. In the following the two terms will be used interchangeably when 
in fact referring to the cost function associated with the minimization problems, which are interesting 
features of this chapter. 


62.1.4.2 Optimal Control Objective and Hamiltonian Functions 
DT 


The objective of optimal controller design is to determine the policy that minimizes the cost (Equa- 
tion 62.5), to obtain 


CO 
V* (x;) = min r(x;, h(x;)) ), (62.8) 
‘ h(.) (x 
which is the optimal cost. The optimal control policy is then 
[o,@) 
h*(x;.) = arg min r(x, h(x;)) }. (62.9) 
k 8 HO) (>: 


A difference equation equivalent to Equation 62.5 is given by 
Vin (Xk) = 1 Xk h(ocK)) + Va%e+1), Vn(0) = 0. (62.10) 


This equation is referred to as Bellman’s equation and is a consistency relation that provides the 
interconnection between the one-step reward function and the cost function associated with a given 
control policy at successive time steps. Bellman’s equation can be solved for the cost function associated 
with any given admissible control policy, instead of evaluating the infinite sum given in Equation 62.5. 
Writing this equation for linear systems with quadratic cost functions one obtains a Lyapunov equation. 
Evaluating the value of a current policy using this equation is a key concept in developing RL techniques. We 
shall show how to solve the Bellman equation on-line in real time using observed data from the system 
trajectories. 

The discrete-time Hamiltonian function can be defined as 


(xg, h(xk), AVE) = 1(xK3 A(xK)) + Valxeg1) — Valxr) (62.11) 


where AVx = Vi (x41) — Va(xq) is the forward difference operator. The Hamiltonian function captures 
the energy content along the trajectories of a system as reflected in the desired optimal performance. The 
Hamiltonian function must be equal to zero for the value associated with a given control policy. We shall 
see later how the discrete-time Hamiltonian, being related to the so called temporal difference (TD) error, 
provides means for online solution for the value function associated with a certain control policy. 


CT 


In the continuous-time case the expressions of the optimal cost and optimal control policy are similar to 
Equations 62.8 and 62.9 and follow directly when the sum operator is replaced by the integral. Similar to 
Equation 62.10, in the continuous-time case one obtains the infinitesimal version of Equation 62.7 


0 = r(x, h(x)) +(VV")" (f(x) +.¢(x)h(x)), V"(0) =0 (62.12) 


where VV" (a column vector) denotes the gradient of the value function V"(x), as the value function 
does not depend explicitly on time. Equation 62.12 is a Lyapunov equation for nonlinear systems which, 
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given the controller h(x) € Y(Q), can be solved for the value function V"(x) associated with it. Given that 
h(x) is an admissible control policy, if V" (x) satisfies (Equation 62.12), with r(x, h(x)) > 0, then V" (x) is 
a Lyapunov function for the system (Equation 62.2) with control policy h(x). 

The Hamiltonian function is defined as 


H(x,h, VV") = r(x, h(x) + (VV")! (f(x) + g(x)h(x)) (62.13) 


and must be equal to zero for the value associated with a given control policy. One can also show that, 
for a given admissible control policy u = h(x), the equation H(x, h, VV") = 0 has the same cost function 
solution as 


t+T 
/ r(x, h(x))dt + V"(x(t + T)) — V"(x(t)) =0, V"(0) =0, (62.14) 
t 
Vx(t) € Q initial condition, and VT > 0, where x(t + T) = g(t + T)= o(t + T; x(t), h(.)). 


62.1.4.3 Bellman’s Optimality Principle 


Bellman’s optimality principle [6], a cornerstone of optimal control, is the central idea of the dynamic 
programming algorithm providing means for the calculation of sequences of optimal control actions (i.e., 
an optimal-action policy). The principle states that an optimal policy has the property that no matter 
what the previous control decisions have been, the remaining decisions determined based on that policy 
will be optimal relative to the state resulting from the previous decisions. 


DT 


Bellman’s optimality principle means that 


V*(x,) = min (r(xp, h(xg)) + V* (e421) - (62.15) 


This equation is known as the Bellman optimality equation, or the discrete-time Hamilton-Jacobi- 
Bellman (HJB) equation. The optimal policy is 


h* (xy) = arg min (r(xp, h(xx)) + V*(xe41)) 5 (62.16) 


Since one must know the optimal policy at time k + 1 to use Equation 62.15 to determine the optimal 
policy at time k, Bellman’s optimality principle yields a backwards-in-time procedure for solving optimal 
control problems (i.e., one has to start from the end-goal and work his way backwards through time to 
determine the optimal control sequence that would provide the smallest cost over the entire time interval). 
For this reason dynamic programming is by nature an off-line planning method that requires the full 
knowledge of the system dynamics. 


CT 


For the continuous-time case the optimal cost function V*(x) satisfies the continuous-time HJB equation 


0O= min [H(x,u, VV*)]. (62.17) 
ueW(Q) 


Assuming that the minimum on the right-hand side of Equation 62.17 exists and is unique, and the 
reward function is given by r(x, u) = Q(x) + u! Ru, then the infinite horizon optimal control solution for 
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the given problem is 
1 
u*(x) = — 5k gi (x)VV*. (62.18) 


Inserting this optimal control policy in the Hamiltonian the continuous-time HJB equation takes the 
form 


0 = Q(x) + (VV*) f(x) — OV TIRE VV" V*(0) =0. (62.19) 


In order to find the optimal control solution for the problem one only needs to solve the HJB equation 
62.19 for the value function and then substitute the solution in Equation 62.18 to obtain the optimal 
control. Solving this nonlinear HJB equation is generally difficult and requires complete knowledge of 
the system dynamics. 

We shall see in the following how Bellman’s equation and optimality principle can be used in ADP 
methods to solve the optimal control problem, both in continuous-time and discrete-time frameworks, in 
an online fashion, based on data measured in real time from the system, in a forward-in-time procedure, 
and, at times, without requiring knowledge on the dynamics of the system to be controlled. 


62.1.5 ADP and Adaptive Critics 
62.1.5.1 ADP 


ADP names the class of RL algorithms that provide in an iterative manner an approximate solution of 
optimal control problems. These methods consist in a series of iterations between the steps of policy 
evaluation/value function update and policy update/improvement. Successful completion of the policy 
evaluation step leads to starting the policy update step. We can say that a policy is an improved one in 
comparison with another one, if it has a smaller associated cost. The iterations end when a policy update 
step no longer leads to an improvement of the control policy. This means that the policy with the smallest 
associated cost, that is, optimal control policy, has been found. 


62.1.5.2 Adaptive Critics 


The structural representation of ADP methods is given by the actor-critic interconnection [5,9,15], 
presented in a general form in Figure 62.1. The actor structure plays the role of the controller, which 
computes the control signal, according to the present control policy, based on the measurements that 
define the present state of the environment/system. The critic structure has the purpose of evaluating 
the performance of the present action policy. Based on the critic’s assessment on the value of the present 
policy, various adaptation schemes can be used to modify the control strategy such that the actor’s new 
policy will yield a value, which is an improvement over the previous one for the entire state space of 
interest (i.e., the new value has to be smaller in the case when minimization of the cost is desired, or 
higher for the case of maximization purposes). 

It is in order here to discuss the concept behind the ADP methods in contrast with the classical 
model-based adaptive optimal control mechanisms. 

In the case of model based adaptive optimal controllers (or indirect adaptive optimal control) 


¢ First an approximation structure, serving as a model of the system, is trained to approximate (i.e., 
learns using an identification procedure) the system dynamics. 

¢ Then this model is used in combination with a controller design method to determine a control 
policy such that the desired optimal performances of the closed-loop system are satisfied. 


These two steps of system identification and optimal controller design can be iterated such that in 
the case in which the system dynamics change, a new control policy can be determined such that the 
desired closed-loop optimal performances can be maintained. One can immediately see that the controller 
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FIGURE 62.1 Actor-critic structure for the implementation of approximate dynamic programming algorithms. 


calculated in this manner brings the closed-loop system close to optimality in direct relation with the 
differences between the approximate model of the system dynamics and the true dynamics of the plant. 
Simply put, the controller is just as “optimal” as the model is “accurate.” 

In the case of ADP algorithms the learning process is moved to a higher level, having no longer as 
object of interest the details of a system’s dynamics but a performance index that quantifies how close to 
optimality the closed-loop control system operates. From this perspective ADP methods or RL in general, 
provide means of learning optimal behaviors (i.e., adaptation of the controller structure to provide an 
optimal control policy) by observing the response from the environment to nonoptimal control policies. 
ADP methods search for the admissible control policy which optimizes the measurable performance. In 
this case the resulting controller is just as “optimal” as the representation of the cost is “accurate.” 


62.1.6 Value Function and Q-Function 


The value function associated with a given control policy has been defined and discussed in Section 62.1.4. 
This is a functional which maps the state space of the system into the space of real numbers. 
Value functions 


¢ Provide the means of comparison between control policies (e.g., in the case of a minimization 
problem, a policy is better than other if it has a smaller value function for all the states in the region 
of interest in the state space). 

« Are required for the calculation of the optimal control policy. 


When solving the minimization in Equation 62.16 one sees that in order to calculate the optimal 
control policy knowledge of the system dynamics is required for representation of x41. For this reason 
such equation can not be used to solve the optimal control problem under a model-free condition. It was 
thus of interest to determine a different function that encapsulates the information of the value while at 
the same time can be minimized with respect to the control policy without maintaining the requirement 
of known system dynamics in a similar fashion to the regular RL methods. 

Since the value function is part of the Hamiltonian function, which is to be minimized when solving 
for the optimal control policy, based on Equation 62.16 a so called Q-function was introduced [23], also 
known as an action dependent value function [24]. As we shall see, the defined Q-function maps the space 
of states and actions into the space of real numbers, and can be minimized with respect to the control 
policy such that it directly provides the optimal controller without necessity of the system dynamics. 
In effect the definition of the Q-function was the key for a completely model-free ADP algorithm, as 
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promised by the RL methods, an algorithm formulated in the regular terms of a standard optimal control 
problem. 


DT 


Starting from the difference Equation 62.10 the Q-function is defined as 


Qn(Xks Uk) = 1(Xks UR) + Va (xe+1) (62.20) 
where xp4.1 =f (xz) + g(xp)ug and uz € R”. The Q-function represents the value associated with using 


¢ The control input u, as one first-step in the control sequence followed by 
¢ The control policy uj, = h(x;) for all the future steps Vi > k+1. 


The relation between the Q-function and the optimal value is given by 
Va) = _ Q* (Xks Uk) (62.21) 
where 
QV (Xks Uk) = 1k» Uk) + V* (xk +1). (62.22) 
One can also write Equation 62.20 only in terms of the Q-function as 


Qh Xk. Uk) = W(X: Uk) + Quek AxK41))- (62.23) 


Equation 62.23, written in terms of the Q-function, is equivalent to Equation 62.10 written in terms 
of the value function. Thus, given a control policy uz = h(x,), Equation 62.23 can be solved for the 
Q-function associated with it. 

In terms of Q* the optimal control is 


(Hr) = arg min (Q" (Xk u))). (62.24) 
UK 
In the absence of control constraints, one obtains the optimal value by solving 
d 2k 
— Q* (xz, u) = 0. (62.25) 
Ou 


CT 


The definition of a Q-function in a continuous-time framework is not straight-forward and will not 
be discussed now. Different definitions were introduced from the perspective of the two equivalent 
Equations 62.12 and 62.14 in [12,14]. 


62.1.7 Key Ingredients for Online Implementation of ADP Methods: 
Value Function Approximation and TD Error 


The key aspects of the online implementation of the ADP algorithms are related to the way in which the 
value function is defined and the manner in which the critic structure learns to approximate the value 
function. This requires answers to the following questions: 


Q1: Which value function does the critic approximate? 
Al: The options are 
¢ Value function 
¢ Action dependent value function (i-e., Q-function) 
¢ Gradient of the value function or gradient of the Q-function. 
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Q2: What is an appropriate parametric representation for the chosen value function? 

A2: In [7,9,24,15] the critic function is represented as a neural network, which is known to have the 

universal approximation property. In this chapter we will discuss the case in which the critic can 
be represented as a linear combination of a set of basis functions, which spans the space of value 
functions to be approximated. 
To motivate this approach, let us determine the value function approximator for the familiar linear 
quadratic regulation (LQR) problem in the discrete-time formulation. In LQR the value associated 
with any admissible control policy uj, = —Kx, is quadratic in the state (ie., Vk (xx) = wee Px, with 
P a positive-definite matrix). Using the Kronecker product notation, the value function can be 
represented as 


Vi (xp) = xf Pax = (vec(P))" (xp @ xz) = p X (62.26) 


where ® denotes the Kronecker product, vec(P) = p is a column vector formed by stacking the 
columns of the matrix P, and x; = x, ® x, is the quadratic polynomial vector containing all possible 
products of the n components of x;.. Noting that P is symmetric and has only n(n + 1) /2 indepen- 
dent elements, one can remove the redundant terms in x, ® x; to define a quadratic basis set x, 
with n (n+ 1)/2 independent elements. The unknown parameter vector, which will adapt during 
learning, is p that contains the elements of matrix P. 


Value Function Approximation 

In the case in which the critic function takes a general nonlinear form, assuming that the value function 
is sufficiently smooth, then, according to the Weierstrass higher-order approximation theorem, there 
exists a dense basis set of functions {¢;(x)} such that the value of any admissible control policy can be 
represented as 


lo) L le) 
Vala) = D> wigi(x) = D> wigile) + D> wigilx) = WT (x) + ex (x) (62.27) 
i=l i=l i=L+1 
with the basis vector (x) = [ 1 (x) 2(x) +: x(x) | :R" —> R" such that W! o(x) is positive- 


definite over the set Q, and where the error ¢,(x) converges uniformly to zero as L — oo. It is standard 
usage to choose a polynomial basis set. Results have been shown for other basis sets including sigmoid, 
hyperbolic tangent, Gaussian radial basis functions, and so on. such that the approximation error €;,(x) is 
bounded by a constant on the compact set (2. One can obtain intuition on choosing the basis set for the 
approximation of the value function knowing that they are positive-definite and they will play the role of 
Lyapunov functions for the closed-loop system. 


62.1.7.1 Q-Function or Action-Dependent Value Function Approximation 


Similar to the value function approximation case, general Q-functions for nonlinear systems can have a 
parametric representation in the form 


Qy(x, u) = W" o(z) + ex(z) (62.28) 


where z? =[x? u!], o(z) is a basis set of activation functions and W is the vector of parameters of the 
Q-function, such that W" o(x, u) is positive definite for all x € Q considering all admissible control 
policies u € (Q). 


Q3: What is the error function which has to be minimized? 
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A3: The error function which is generally minimized during the Critic learning phase of an ADP algo- 
rithm is the TD error defined as 


ee = 1 (xp, A(x) + Va xnta) — Valen). (62.29) 


One sees that the right-hand side of this equation is the discrete-time Hamiltonian. Also, if Bellman’s 
equation holds then the TD error is zero. Therefore, to find the value V;,(.) of a fixed control policy 
ux = h(x,) one can solve the equation e, = 0, for all x, € Q. 

Q4: Which identification method is employed for determining online the parameters of the approximate 
representation of the value function? 

A4: To answer this question we consider the discrete-time and continuous-time cases separately. 


DT 


Considering the chosen parametric representation for the value function one can write the TD error 
equation in the form 


eg = r(xg, W(xg)) + Wi b(xn41) — Wi o(xp)- (62.30) 


a. Based on measurements of the states at successive moments in time x;, x44, and of the rewards 

r(xz, h(x,)), taken from the system/environment while using the control policy uz = h(x,), one 
can set up an online parameter identification procedure, of the sort described in this handbook, 
to solve for the parameters W), of the cost function associated with the given control policy such 
that the TD error is minimized in the least-squares sense over the entire set Q. 
Solving on-line for the parameters of the value function is equivalent to obtaining on-line an 
approximate solution of a nonlinear Lyapunov equation. The solution is obtained in this case 
without using knowledge on the dynamics of the system, while only using data measured along 
the state trajectories produced by the specified admissible control policy. 

b. Another approach to the solution starts with the observation that Equation 62.10 is a fixed point 
equation. The solution of this equation can then be reached by means of the contraction 


Vi (xg) = (eps A(xq)) + VF" eta)» (62.31) 


starting with an initial guess VG. As j — oo then Vi) — V;,(.) uniformly over Q. Based on this, 
in terms of the parametric approximation of the value function, the approximate solution of e, = 0 
can be obtained by solving iteratively for the parameters Wj, for j > 1, starting with an initial guess 
Wo. At each iteration step j, the error 


ex(Wj) = r(xes h(xe)) + Wi (xk 41) — Wi) b(ae) (62.32) 


has to be minimized in the least-squares sense while making use of online measured data. 
The error e,( Wj) is called TD prediction error as it can be interpreted as an error between 
e The new overall value represented as Wi (xx). 


« The predicted performance corrected by observation r(x;, h(x,)) + Wie o(%R+1) obtained 
in response to an action applied to the system. 
The TD error expressed in terms of the Q-function approximation is 


ex(W) = r(xp, h(xg)) + We b(ze41) — Wz). (62.33) 


This equation can be solved online, in a similar fashion as described above, for the values of the 
parameters of the Q-function. 
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We now discuss a potential problem that appears when solving on-line for the parameters of the 
Q-function, and then we provide the solution. To make the issue clear, let us take again, as example, the 
familiar LQR case. 

The reinforcement function has the form r(x;z, ug) = ce Qaxk + uf Ru. The Q-function associated with 
a control policy uz = h(x,) is 


Ques UK) = xp Qaxe + ug Rug + XK Prxk+1 (62.34) 
and inserting the system dynamics equation x,41 = Ax, + Bu; one obtains 
Qh (xk; Uk) = x} Qaxk + uj Rug + (Axe + Bux)! Py(Axg + Bux) (62.35) 


which can be arranged in the form 


T T T 
__ | Xk QgtA*PrA A’ P,B Xk 
Qh ks Uk) = *| | BTP,A = R+B™P,BI | uz Cze0) 
Then the Q-function will be written as 

| THe, EG | 

Fr k Xx xu k 
Xk, UK) = Z, Hz, = 62.37 
Qh (xk, Uk) = 2 Azz fl fe | | ( ) 


where Hy, = Qa +A! PA, Hey = A’ PB, Hyx = B'P,A and Hy, = R+B' PB. 

In this case, when it comes to learning online the parameters of the Q-function based on measured 
data from the system, one encounters a problem: Due to the fact that the control policy signal uj, = Kx, 
is a linear combination of the values of the state, the persistence of excitation condition required for the 
regression vector ~(zx) — (z+) does no longer hold. 

This problem is solved by adding persistently an exciting probing noise to the control inputs that were 
calculated based on the state-feedback control policy [8]. In this case the control inputs that are sent to 
the system during the online identification procedure of the Q-function are uz = Kx, + nx. In [1] it is 
shown that this does not result in any bias in the Q-function estimates. 


CT 


The value function associated with the performance of a continuous-time state-feedback control policy 
can be similarly determined using a parameter identification procedure. The only difference consists in 
the fact that the one-step reinforcement signal r(x,, h(x;,)), measured from the system in the discrete-time 
case, is now replaced by the measurement of the integral term J, an r(x(t), h(x(t)))dt. 

The TD error is defined in this case as 


+T 
e(x(t), T, h) = i r(x, h(x))dt + V"(x(t + T)) — V"(x(t)). (62.38) 
t 


Q5: When is the value function identification procedure (i.e., policy evaluation step) finished, such that 
the policy update step can be started? 

AS: The value function identification procedure ends when the convergence of the critic parameters has 
been obtained, in the sense that the predictions of the critic match closely, within some bounds that 
depend on the selected model, number of parameters to be trained and measurement errors, the 
measurements of the reinforcement received from the environment. 

Q6: Is the online identification of the value function always possible? 

A6: All ADP algorithms are developed on the assumption that correct estimation of the value function 
is possible. This means that, in order to successfully apply the online learning algorithm, enough 
excitation must be present in the system to guarantee correct estimation of the value function at 
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each value update step. In relation to this, one must also note that if the policies obtained at the 
policy update steps are admissible policies then they may not provide the necessary excitation for 
the success of the value function learning phase. This is the well-known exploration/exploitation 
dilemma [17], which characterizes adaptive controllers that have simultaneous conflicting goals 
such as optimal control and fast and effective adaptation/learning. Solutions to this problem can be 
imagined (e.g., adding persistently exciting components to the signals computer by the controller). 


62.1.8 Policy Update 
DT 


Given the value function associated with an admissible control policy, say V;(x;) is the value associated 
with the policy u, = h(x;), then the update of the control policy is given by the equation 


wrx) = arg min. (r(x4, voxR)) + Vilsie1))- (62.39) 


The new resulting control policy will be improved compared with the old one in the sense that it will 
have a smaller associated value. 
For the case in which the Critic learns the Q-function, the policy update step is based on 


ce) 
— Q(x, u) = 0. (62.40) 
Ou 
Using the Q-function approximation, Equation 62.40 becomes 
~ atx u) = woe, u) = 0. (62.41) 
Since the Q-function depends explicitly on the control action u, the derivatives can be computed 
without reference to further details such as the system dynamics. Thus, in order to obtain an explicit 
updated policy uz = h(x,) one requires application of the implicit function theorem to the Q-function 
approximation structure. 
For the LQR case the Q-function is given by Equation 62.37 and the policy update step becomes 


up = — (Huu) HuxXt- (62.42) 


From this equation one sees that in this case, since the kernel matrix H has been found using online 
learning, knowledge on the system dynamics is not needed for this policy improvement step. 


CT 


In the continuous-time case the policy update is performed according to the equation 


=arg min [H(x,v, VV")]. (62.43) 
vew(Q) 


Considering the approximate representation of the value function V"(x) = W" (x) associated with an 
admissible control policy u = h(x), and the particular form of the reinforcement r(x, u) = Q(x) + u! Ru, 
one obtains explicitly the updated control policy as 


w(x) = —R-IgT (x 2 (Be) W. (62.44) 
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62.1.9 Features Which Provide Differentiation between ADP Algorithms 
The differences between the ADP algorithms are related to 


¢ The performance function, which has to be approximated through learning by the critic structure 
¢ Value function: The algorithms which use the value function as means of evaluation of the 
control performance are also referred to as V-learning. These are the default case of ADP. 
¢ Action dependent value function (i.e., Q-function): In this case the algorithms are referred to as 
Q-learning or action dependent-learning. Due to the virtues of the Q-function such algorithms 
have the desired model-free property. 
¢ Gradient of the value function: In this case the algorithms are referred to as Dual-learning. 
¢ Gradient of the Q-function: These ADP are known as action dependent dual-learning. 
¢ The conditions in which the control policy is updated 
¢ Thus, if the policy update is performed before the critic’s parameters have converged to the 
values associated with the value of the present control policy then we obtain value iteration or 
heuristic dynamic programming (HDP) algorithms. 
¢ Ifthe policy update is performed after convergence of the critic’s parameters has been obtained 
such that the critic function represents the value associated with the current control policy we 
obtain policy iteration algorithms. 


62.1.10 Terminology—Revisited 


Reinforcement learning: Class of methods that provide solution, in an online fashion, to optimal control 
problems by means of a reinforcement scalar signal measured from the environment, which indicates the 
level of control performance. 

Behavior policy: The sum of all rules that are followed by an active agent while interacting with its 
environment. 

Control policy: A function which maps the state space of the system to be controlled onto the space of 
control actions that can be applied to the system. 

Value function: A function which maps the state space of a system onto the set of real numbers 
such that it represents the amount of rewards obtained in response to performed control actions, 
while starting from a given initial state in the state space. Value functions provide an index of per- 
formance of a certain control policy. Control policies that have higher value functions are better 
than others. The term is used in relation to performance functions associated with maximization 
problems. 

Cost function: A function which maps the state space of a system onto the set of real numbers, such 
that it reflects the amount of resources used for control purposes when starting from a specific state 
of the system. Control policies with smaller costs are better than others. As minimization problems are 
converse to maximization problems, with slight abuse of language cost functions are also referred to as 
value functions. 

Q-function: A function which maps the space of states of a system and actions that can be performed, 
onto the space of real numbers such that it reflects the value associated with using an initial action (located 
in the action space) at a specific initial state of the system followed by optimal actions for all the subsequent 
control steps. 

ADP: Class of algorithms which provide online solution to optimal control problems by using approxi- 
mate representations of the value function to be minimized and of the control algorithm to be performed, 
and employing Bellman’s optimality principle, central in dynamic programming, to provide means for 
training on-line the two approximation structures based on measured data from the system. Being math- 
ematically formulated, such algorithms allow development of rigorous proofs of convergence for the 
approximation based approaches. 
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Actor-critic structure: The structural representation of ADP algorithms. It reflects the informational 
interconnection between 


¢ The actor, which reacts in real-time to measurements from the system, and learns to adapt based 
on performance information from the critic. 

¢ The critic, which learns to approximate a value function based on performance data and state data 
measured from the system, and provides performance information relative to the presently used 
control policy to the actor. 


Adaptive critics: All algorithms which provide means for learning optimal control policies in an online 
fashion while using an actor-critic structure. 

Bellman’s optimality principle: The central idea of the dynamic programming algorithm, which states 
that an optimal policy has the property that no matter what the previous control decisions have been, the 
remaining decisions determined based on that policy will be optimal, relative to the state resulting from 
the previous decisions. 

Bellman’s equation: The mathematical expression of the relation between the one-step reward function 
and the infinite horizon cost function associated with a given control policy. 

Value function approximation: The class of techniques used for online calculation of the 
parameters of an approximate expression of the value function associated with a given control 
policy. 

TD error: The error difference between 


¢ The expected infinite horizon cost predicted at the previous time step. 

¢ The summation between the predicted future cost at the present time and the obtained reward 
over the time interval between the previous time step and the present time moment (i.e., prediction 
corrected by observation). 


62.2 ADP Algorithms for Discrete-Time Systems 


In this section are outlined the ADP algorithms developed considering the discrete-time formulation of 
the system to be controlled, of the control policy to be learned, and of the reinforcement signal, which 
provides means for critic learning. The connection between the ADP algorithms and optimal control is 
given at the end of this section, where we provide the equivalent algorithms for the well-known LQR 
problem in terms of iterations on discrete-time algebraic Riccati equations. 


62.2.1 Policy Iteration 


The policy iteration technique is applicable to a broad class of systems and it is guaranteed to converge 
to the optimal control with stepwise stability. The convergence guarantee is maintained also for the ADP 
online version of the policy iteration technique, which uses a value function approximator in the form of 
the critic structure. 


62.2.1.1 V-Learning 
62.2.1.1.1 V-Learning Policy Iteration Algorithm 


Initialize: Select any admissible control policy ho(x;) and let j = 0. 
Policy Evaluation Step: Determine the value of the current policy using Bellman’s equation 


Vp, (Xk) = 1 (Xk Hy (Xk) + Vn; e+1) (62.45) 


62-16 Control System Advanced Methods 


Policy Improvement Step: Determine an improved policy using 
hea x) = arg min (xp h(a) + Vi (441) (62.46) 
Stop Algorithm Step: If 
||hj+i(xe) — hj (xn) || <es (62.47) 


then STOP, else let j = j + 1 and go to the policy evaluation step. 


As j > ©, this algorithm converges to the optimal value (Equation 62.8) and optimal control policy 
(Equation 62.9). The algorithm can be stopped when the norm of the difference between the two successive 
controllers is smaller than a specified error ¢,. This means that an approximate optimal control policy has 
been determined. 

If the reinforcement is given in the special form r(x,, uz) = Q(x~) + uj Rug then the policy improve- 
ment step becomes explicitly 


hjsi(xg) = —R7'g? (x4) V Vig (Xk41) (62.48) 


where V V(x) = (OV(x)/0x), a column vector, is the gradient of the value function. 
The online version of this policy iteration algorithm implemented using value function approximation 
on an actor-critic structure is now given. 


62.2.1.1.2 Online V-Learning Policy Iteration Algorithm 


Initialize: Select any admissible control policy ho(x;) and let j = 0. 
Policy Evaluation Step: Determine online the parameters of the critic Wj such that 


W;) (Cx) — b(xK41)) = ree, (XK) (62.49) 


is solved in the least-squares sense. 
Policy Improvement Step: Determine an improved policy using 


hea ex) = ang min (1x4, h(a) + WH o(xK41)). (62.50) 


Stop Algorithm Step: If 
|| Wj — Wj-1|| < ews (62.51) 
then STOP, else let j = j + 1 and go to the policy evaluation step. 


The algorithm can be stopped when the norm of the difference between the two successive value of the 
critic parameters is smaller than a specified error ews. 

Any online parameter identification algorithm can be used for the policy evaluation step such that the 
critic parameters can be determined, in the least-squares sense, based on online measured data of the 
system states x;,xx41 and the reward r(xx, hj(x;)). The convergence of online parameter identification 
algorithm requires that the regression vector (¢(x;) — (xx41)) is persistently exciting. 

If the reward function has the form r(x,, uz) = Q(x~) + ug Rug then the improved policy is given by 


1 
his (%k) = —FR-18' (a) VO Crees) Wy (62.52) 


where V(x) = (06(x)/Ox) € REX" is the Jacobian of the vector of basis functions. 

One sees that even if the policy evaluation step can be executed without using knowledge on the system 
dynamics, based only on measurements form the system, the policy improvement step of the algorithm 
requires complete knowledge on the system dynamics since they appear in the expression of x;.41, that is, 
Xap =f (xk) + g (xp) A(x). 
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62.2.1.2 Q-Learning 


The policy iteration algorithm on the Q-function has been introduced in [8] to solve the LQR problem. 


62.2.1.2.1 Q-Learning Policy Iteration Algorithm 


Initialize: Select any admissible control policy ho(x;) and let j = 0. 
Policy Evaluation Step: Determine the value of the current policy using Bellman’s equation 


Qh; (Zk) = 1 (Xk hy (*k)) + Qn; (Zke+1) (62.53) 


Policy Improvement Step: Determine an improved policy using 
hye (ex) = argmin (Cenerer))) (62.54) 
Stop Algorithm Step: If 
||hj+1(xe) — hy (xn) || <es (62.55) 
then STOP, else let j = j + 1 and go to the policy evaluation step. 


62.2.1.2.2 Online Q-Learning Policy Iteration Algorithm 


Initialize: Select any admissible control policy ho(x;) and let j = 0. 
Policy Evaluation Step: Determine the parameters of the critic Wj, by solving online, in the 
least-squares sense, the equation 


W;) ((zk) — b(Ze4.1)) = 1 %ks hy (xK)). (62.56) 


Policy Improvement Step: Determine an improved policy using 
hea (ox) = arg min (W/ 60x51) (62.57) 
Stop Algorithm Step: If 
|Wi— Wal] < ews (62.58) 


then STOP, else let j = j + 1 and go to the policy evaluation step. 


Any online parameter identification algorithm can be used for the policy evaluation step such that the 
critic parameters can be determined, in the least-squares sense, based on online measured data of the 
system states xx, xx41 and the reward r(xx, hj(xx)). 

In contrast to the online V-learning algorithm, for the online Q-learning algorithm the policy update 
step can be executed without using any knowledge on the system dynamics. This is by virtue of the 
formulation of the Q-function. One sees now that the policy iteration algorithm where the Critic structure 
learns the Q-function can be executed without using knowledge on the system dynamics. This means 
that it is a completely model-free method, which provides online, in a stepwise adaptive manner, with 
guaranteed stepwise stability, an approximate optimal controller. 


62.2.2 Value Iteration 


Here we present value iteration algorithms. The convergence of value iteration algorithms, while using 
linear systems with quadratic cost indices, has been proven in [11]. In [1] the convergence of these 
algorithms was shown while solving H-infinity problems. The convergence proof for nonlinear value 
iteration was given in [2]. 
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62.2.2.1 V-Learning 
62.2.2.1.1 Value Iteration Algorithm (HDP) 


Initialize: Select any control policy ho(x;), not necessarily admissible or stabilizing, and Vo(x,) such 
that Vo(0) = 0, and let j = 0. 
Value Update Step: Update the value using 


Visas (xp) = (xp, hig) + Vira): (62.59) 


Policy Improvement Step: Determine an improved policy using 


hea (sk) = arg min (r(x, M(ax)) + Vier ee) (62.60) 
Stop Algorithm Step: If 
I hj+1 x) — hjlxe) || <es (62.61) 


then STOP, else let j = j + 1 and go to the value update step. 


It is important to note that the value iteration algorithm does not require an initial stabilizing policy. 

Writing together the two steps of the value iteration algorithm one obtains a recursion on Bellman’s 
optimality equation. This shows that the convergence of the value iteration algorithm is based on the fact 
that Bellman’s optimality equation is a fixed point equation. The interleaved steps of value update and 
policy improvement are the means of using the contraction map such that the fixed point solution of this 
map, that is, the optimal cost function, is obtained iteratively. For the case of the LQR problem the value 
iteration algorithm is equivalent with iterations on the discrete-time algebraic Riccati equation. 


62.2.2.1.2 Online Value Iteration Algorithm 


Initialize: Select Wo and any control policy ho(x;), not necessarily admissible or stabilizing, and let 
j=0. 
Value Update Step: Determine online the parameters of the critic Wj such that 


Wi (xx) = (xe, hy(xx)) + Wi bey): (62.62) 


is solved in the least-squares sense. 
Policy Improvement Step: Determine an improved policy using 


hea ex) = arg min (roe, AC) + WH O(%e1))- (62.63) 
Stop Algorithm Step: If 
|| Wi+1 — Wj < ews (62.64) 


then STOP, else let j = j + 1 and go to the value update step. 


If the reward function has the form r(xz, uz) = Q(x~) + uj Rug then the improved policy is given by 


1 
hyp) = — SR gh (x)VO! Ce) Wi (62.65) 


where V(x) = (06(x)/Ox) € REX" is the Jacobian of the vector of basis functions. 

Note that, when solving the value update step, the regression vector is (x;), which must be persis- 
tently exciting for convergence to the least-squares to be achieved when using a recursive method of 
identification. 
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62.2.2.2 Q-Learning 
Value Iteration Algorithm (Action Dependent Heuristic Dynamic Programming): 


In a similar manner as for the HDP method, it can be derived an action-dependent heuristic dynamic 
programming (ADHDP) method, in which the critic learns the Q-function, with the fixed point equation 


Qh xg, h(xg)) = 1(xK, A(xK)) + Qn(xp+1 ACK 41))- (62.66) 


62.2.2.3 Dual Learning and Dual Action-Dependent Learning 


In [24] it is shown how RL techniques can be used while the critic evaluates the costate function 


- OVi(Xk) 


nN 
Ox 


(62.67) 
which is the gradient of the value function. This function carries more information about the cost than 
the value function alone. The ADP algorithms where the critic learns the costate function are called dual 
heuristic programming (DHP). 

The fixed point equation which allows implementation of value iteration on the dual value function is 


fe) 0 0 
— V, = — wh — V, 62.68 
Ox n(Xk) Ong r (xz, A(x) + Dn n(Xk41) ( ) 


or, explicitly, 


(62.69) 


Or (xz, Uu Ou T Or(xp,u Ox Ox, du, 17 
a (xk, UK) re (xk, Uk) 4 | Ota eh OE Ne 
Oxk Oxk Ouk Oxk Oug Oz 


for a prescribed policy uz = h(x). A critic structure can be used to approximate d; and online learning 
can be implemented is a similar fashion as discussed before. 

Since in Equation 62.69 (0x41 /Oxx) = f (xq) and (Ox;41/Oux) = g (xx), one sees that any ADP scheme 
based on this fixed point equation requires knowledge of the full plant dynamics. Moreover, it is clear that 
in this case the online implementation of the value function identification techniques is computationally 
intensive as the costate function is a vector. 

Similarly, there have been formulated algorithms based on the gradient of the Q-function. The resulting 
ADDHP algorithm has the same features noted for DHP. 


62.2.3 Policy Iterations and Value Iterations on the Algebraic Riccati 
Equation of the LQR Problem 


We have seen how to implement V-learning and Q-learning on-line for nonlinear systems using value 
function approximators. In the case of Q-learning, no knowledge of the system dynamics is needed for 
online RL and convergence to the optimal control solution. In this subsection we derive the equivalent 
algorithms, which provide solution for the discrete-time LQR problem, in terms of standard ideas from 
systems theory. The algorithms presented next are underlying the ADP methods, which have just been 
discussed. The reader is, however, advised that these algorithms are not for implementation purposes, 
since they require knowledge of the full system dynamics. 


62.2.3.1 Discrete-Time LQR Problem 


The system dynamics are given by 
Xko1 = Ax, + Buy (62.70) 


where x, € R”, uz, € R”, A € R"*", and Be R"*™. 
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The objective is to calculate the optimal state-feedback control policy uj, = K*x,, which minimizes the 
infinite horizon quadratic cost functional 


lo) 
V (xn) = Do x} Quai t+ uj Ruj (62.71) 
i=k 


where Qg € R”*”", Qg > 0, and RE R”™*”, R>O0. 
It is known that the optimal cost is quadratic in the state and is given by 


V* (xg) = xp Px (62.72) 
with P > 0. 
62.2.3.2 V-Learning Policy Iteration 


Algorithm 


Initialize: Select any admissible control policy uz = ho(xz) = Kox, and let j = 0. 
Policy Evaluation Step: Determine the value of the current policy, that is, the value of the matrix Py, 
using Bellman’s equation 


xp Py Xk = x} Qaxk + uj Rug + Xp PhjXk-1- (62.73) 
Equation 62.73 is explicitly 
xp PnjXk = X_ Qaxk + uy Rug +4 (A+ BKj)’ Py, (A + BK) xx (62.74) 
or, simply the discrete-time Lyapunov equation 
Phy, = Qa + Kj’ RKj + (A + BKj)"P}, (A + BKj). (62.75) 
Policy Improvement Step: Determine an improved policy using 
Kjx1 = arg min (Qu +K™RK + (A+ BK)" Py, (A+ BK)) (62.76) 
which explicitly is 
Kjy1 =—(R+ B" PB) 'B" Py, A. (62.77) 
Stop Algorithm Step: If 
|| Kina — Kjl| < exs (62.78) 
then STOP, else let j = j + 1 and go to the policy evaluation step. 


62.2.3.3 V-Learning Value Iteration 
Algorithm 


Initialize: Select any control policy uz = ho (xz) = Kox, and Po > 0, and let j = 0. 
Value Update Step: Update the value, that is, determine the value of the matrix P;+1, using 


xg Pip Xp = xf Qaxk + uj Rug + xp Pixt.- (62.79) 
which is explicitly 


Pix1 = Qa + Kj RK; + (A+ BKj)"P;(A + BK;). (62.80) 
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Policy Improvement Step: Determine an improved policy using 
Kjsi = arg min (Qu +K™RK + (A+ BK)" Pi (At BK)) (62.81) 
which explicitly is 
Kj41 = —(R+ B"Pj41B) 1B! Pi A. (62.82) 
Stop Algorithm Step: If 
| Kit — Kjl| < exs (62.83) 


then STOP, else let j = j + 1 and go to the policy evaluation step. 


The test (Equation 62.83) can be replaced by the test ||P —P; | < €ps; which is in terms of the value 
function. 


62.2.3.4 Q-Learning Policy Iteration 
Algorithm 


Initialize: Select any admissible control policy uz = ho(xz) = Kox, and let j = 0. 
Policy Evaluation Step: Determine the value of the current policy, that is, the value of the matrix Hy, 
using Bellman’s equation 


SD) ESI-[T (8 ele} Lec) Le] em 
Uk 1 | UK Uk O RI Lug KjxK41 7 LKjxky1 
or, the discrete-time Lyapunov equation 
Q@ O),fA B], [A B 
y= [0 altlca gal Lge Kal si 
Policy Improvement Step: Determine an improved policy using 
iin ; xe] Xk 
j+1 = arg min (ea An, i) (62.86) 
which explicitly is 
Kj41 = — (Haut) Hust (62.87) 


Stop Algorithm Step: If 
| Kivi — Kj] < exs (62.88) 
then STOP, else let j = j + 1 and go to the policy evaluation step. 


62.2.3.5 Q-Learning Value Iteration 
Algorithm 


Initialize: Select any control policy uz = ho(x;) = Kox, and Ho > 0, and let j = 0. 
Value Update Step: Update the value, that is, determine the value of the matrix Hj41, using 


ff 
uz [Qe D8 AB [AB 
in =[° RIT KA KB Hj KA KpB| (62.89) 
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Policy Improvement Step: Determine an improved policy using 
Kjy1 = —(Hyuj+1) Hux jet (62.90) 
Stop Algorithm Step: If 
| Kix — Kjl| < exs (62.91) 
then STOP, else let j = j + 1 and go to the policy evaluation step. 


The test (Equation 62.83) can be replaced by | Aj41 — Aj | < €ys, which is in terms of the parameters 
of the Q-function. 


62.3 ADP Algorithms for Continuous-Time Systems 


Similar to the ADP methods presented above, which were developed considering discrete-time system 
dynamics, there have also been developed algorithms, which aim at solving the optimal control problem 
for systems with continuous-time dynamics. This is strongly motivated by the fact that the dynamics of 
a large class of human-engineered systems unfold in continuous-time and an approximate discrete-time 
formulation, obtained by discretization techniques, would only result in suboptimal control strategies for 
the considered systems. The formulation of continuous-time ADP methods starts from the fact that the 
infinite horizon cost associated with the use of a certain control policy u = h(x) can be written as 


t+T 
V"(x(t)) = i r(x(t), h(x(t)))dt + V" (x(t +T)), (62.92) 
t 


for any time interval T > 0. This equation has the same value function solution as 
0=H(x,h, VV"). (62.93) 


It has been shown in [18] that Equation 62.92 is a fixed point equation for continuous-time systems, 
similar with Bellman’s equation defined for the discrete-time case. 
According to Bellman’s optimality principle, the optimal cost satisfies 


t+T 
V*(x(t)) = min (/ r(x(t), u(t))dt + V* (x(t + 7) ; (62.94) 
u(t); t<t<t+T t 
while the HJB equation is 
O0= min [H(x,u, VV*)]z. (62.95) 
ueW(Q) 


The optimal control policy is then given by 


+T 
h* (x(t)) = arg min ([ r(x(t), h(x(t)))dt + V* (x(t + a) . (62.96) 
x) \Jt 


In the continuous-time case the TD error over any interval T > 0 is defined as 


+T 
e(x(t), T,h) = i r(x, h(x))dt + V(x(t + T)) — V(x(t)). (62.97) 
t 


It is now direct to formulate the policy iteration and value iteration algorithms for continuous-time 
systems given in [19,20]. In Section 62.3.3 we provide the connection between these algorithms and 
classical results from optimal control theory by presenting their underlying formulation while solving the 
well-known continuous-time LQR problem. 
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62.3.1 Policy Iteration 
Algorithm 


Initialize: Select any admissible control policy h© (x) and let j = 0. 
Policy Evaluation Step: Solve for yr? (x(t)) using 


v" (x(t) = / eC, HO Gayde + VEE TY) with V0) = 0 
t 
Policy Improvement Step: Determine an improved policy using 
AUTD — arg min[H(x, u, vey) 
Stop Algorithm Step: If 
| H+) —APe9] <e, 


then STOP, else let j = j + 1 and go to the policy evaluation step. 


The algorithm converges to the optimal controller as j + oo. 
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(62.98) 


(62.99) 


(62.100) 


For the special case in which the instantaneous reward function is given by r(x, u) = Q(x) + u? Ru and 


the system is Equation 62.2 then the improved state-feedback control policy is explicitly written as 


iT 
AUT) (x) = SR lg"(xvvi 


Online Policy Iteration Algorithm 


Initialize: Select any admissible control policy h© (x) and let j = 0. 
Policy Evaluation Step: Solve online for the parameters of the critic W; such that 


+T 
W;'Lo(x(¢)) — o(x(t + T))] = i Hao), A(x e)))de_ where (0) = 0 
t 


is solved in the least-squares sense. 
Policy Improvement Step: Determine an improved policy using 


Ab(x)\" 
AUD (x) = arg min [ (. u, ( i ) ")| 


| Wj — Will < ews 


Stop Algorithm Step: If 


then STOP, else let j = j + 1 and go to the policy evaluation step. 
If r(x, uw) = Q(x) + u? Ru then the policy improvement step becomes 


db(x)\" 
Ox ) 


Wj. 


het) (x) = = FR 1gT (x) ( 


(62.101) 


(62.102) 


(62.103) 


(62.104) 


(62.105) 


Policy iteration becomes Newton’s method [10] for solving the continuous-time algebraic Riccati 


equation associated with finding the solution of the LQR problem. One can now see that, using the 


RL approach, Newton’s algorithm can be implemented using only information about the input to state 


dynamics of the system. That is, continuous-time policy iteration on the value function solves the con- 


tinuous time algebraic Riccati equation without knowing the system internal dynamics by using data 


measured online alongthe system’s state trajectories. 
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62.3.2 Value Iteration 
Algorithm 


Initialize: Select any control policy h© (x), not necessarily stabilizing, V°(x) such that V°(0) = 0, and 
let j = 0. 
Value Update Step: Solve for V/*! using 


vitFl(x(¢)) = i aoe r(x(s), h (x(s)))ds + Vi (x(t + T)) with V/+1(0) = 0 (62.106) 
t 
Policy Improvement Step: Determine an improved policy using 
AUT) — arg min[H (x, u, VVIt1)] (62.107) 
Stop Algorithm Step: If 
| AU+D (x) — AD (x) | Si (62.108) 
then STOP, else let j = j + 1 and go to the value update step. 
In the case of r(x, u) = Q(x) + u? Ru the policy improvement step is explicitly 
AFD (x) = — a gi (x)VVItI, (62.109) 


Online Value Iteration Algorithm 


Initialize: Select any control policy h(x), not necessarily stabilizing, any vector of parameters Wo, 
and let j = 0. 
Value Update Step: Solve on-line for the parameters of the critic, Wj+1, such that 


+T 
Wi (x(t) = / r(x(s), AY (x(s)))ds + W;' o(x(t + T)) with (0) = 0 (62.110) 
t 


is satisfied in the least-squares sense. 
Policy Improvement Step: Determine an improved policy using 


; a T 
nt) — arg min[H (1 ( a) Wer) (62.111) 
Stop Algorithm Step: If 
|| Win — Will < ews (62.112) 


then STOP, else let j = j + 1 and go to the value update step. 


It is observed that both continuous-time versions of the policy iteration and the value iteration do not 
require knowledge about the internal dynamics of the system, that is, the function f(x). Both these 
algorithms can be implemented online using online identification techniques for the parameters of 
the control policy (i.e., actor) and value function (ie., critic) structures. The actor-critic structure for 
the implementation of these algorithms is discussed in the next section. 
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Critic 


=. 1 Cost function V(x) 
i i 


Controller 
L(x) 
r' 


Sytem 
x = f(x) + g(x)us xq 


FIGURE 62.2 Structure of the system with adaptive controller for the implementation of continuous-time ADP 
methods. 


62.3.3 Actor—Critic Structure for the Continuous-Time ADP Algorithms 


We conclude the presentation of ADP algorithms for continuous-time systems with a discussion of the 
actor-critic structure used in this case. The structure of the system with optimal adaptive controller is 
presented in Figure 62.2. 

The ADP algorithms require only measurements of the states at discrete moments in time and knowl- 
edge of the observed value function over the interval [t, f+ T]. We denote with 


t+T 
d(t, T, u(.)) = i r(x(t), u(x(t)))dt (62.113) 
t 


the reinforcement signal measured over the time interval T. Thus, the data measured at each time 
increment, which is required for the critic structure to learn the value function, is formed by the sets of 
triplets (x(t), x(t + T), d(t, T, u(.))). Since the fixed point equations involved in the formulation of the 
continuous-time ADP algorithms are valid for any positive value of the time interval T, the reinforcement 
time interval T need not have the same value at every step of the iteration. The value of T can be adapted 
online in relation to the amount of time required to measure meaningful information from the system. 

In order to extract reward information regarding the cost associated with the given policy the system 
was augmented with an additional state V(t), with dynamics V = Q(x) +u! Ru. The update of both the 
actor and the critic is performed at discrete moments in time based on sampled information, while the 
actor performs continuous-time state-feedback control. Since the critic learns the value function based on 
observations of the continuous-time value over a finite interval, the algorithm converges to the solution 
of the continuous-time optimal control problem. 


62.3.4 Policy Iterations and Value Iterations and the LQR Problem 


We present next, in terms of standard ideas from optimal control theory, the iterative algorithms equiv- 
alent with the online ADP methods, which provide solution for the continuous-time LQR problem. 
We must again specify that these algorithms, which underlie the iterative ADP methods, are not for 
implementation purposes, since they require knowledge of the full system dynamics. 


62.3.4.1 Continuous-Time LQR Problem 


The system dynamics are given by 
x = Ax+ Bu (62.114) 


where x € R", u € R™, A € R”*", and B Ec R”™”™. 


62-26 Control System Advanced Methods 


It is desired to determine the optimal state-feedback control policy u = K*x, which minimizes the 
infinite horizon quadratic cost functional 


V(x(t), u(.)) = i Fe (x? Qx + u? Ru)dt (62.115) 
t 


where Qe R”*”", Q>0,and Re R”*™", R>0O. 
It is known that the optimal cost is quadratic in the state and is given by 


V*(x(t)) = x(t)! P* x(t) (62.116) 
with P* > 0. 


62.3.4.2 Policy Iteration 
Algorithm 


Initialize: Select any admissible control policy h (x) = K°x and let j = 0. 
Policy Evaluation Step: Solve for yh? (x(t)) using 


. t+T . ’ 
ve (x(t)) = / (x7 Qx + APT (x)RAP(x))\dt + VO” (x(t + T)) with V" (0) =0 (62.117) 
t 


which can be written as 
t+T d(x? ph? x 
/ dt 


Taking the derivative and using the system dynamics in (62.118) it becomes the Lyapunov equation 


) t+T ; F 
dt= / [x?(Q+ KI" RK!)x]dt (62.118) 
t 


(A+ BKA)T ph? 4 ph? (a 4 BKI) = —(Q+ KITRKI) 


which can be solved for Ph”. 
Policy Improvement Step: Determine an improved policy using 


hU+Y(¢) = —Ro!BT ph x (62.119) 
Stop Algorithm Step: If 
AV) —APEQ| <e, (62.120) 
then STOP, else let j = j + 1 and go to the policy evaluation step. 
Initialize: Select any control policy h© (x) = K°x and Pp > 0, and let j = 0. 


Value Update Step: Update the value, that is, determine the value of the matrix P;;1, using 


x" (t)Pi41x(t) = / ee [x7 (Q+ KIT RK) x]dt + x" (t+ T)Pjx(t + T). (62.121) 
t 
Differentiating Equation 62.121 with respect to time along the trajectories given by the controller 
‘9 (x) = Kix one obtains the Lyapunov equation 
(A+ BR!)"Pi41 + Pi: (A+ BK!) + KIRK) + Q= (et 85)")7 (4 + BK)" P; + P(A + BR!) 
+ KIT RK) + Q)eAtBRIT, (62.122) 
Adding and subtracting A/ P; + P;A; and using the notation 
Ric(P;) = A’ Pj + P,A+ Q— P;BR'B' P; (62.123) 


Equation 62.122 becomes 


Al (Pitt — Pi) + (Pit — Pi) Ai = —Ric(P;) + eA Ric(Pie4?. (62.124) 
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Policy Improvement Step: Determine an improved policy using 
AUD (x) = KIt1x = —R7'BT Pix. (62.125) 
Stop Algorithm Step: If 
Ki! — KI || < ex; (62.126) 
then STOP, else let j = j + 1 and go to the value update step. 


The test (Equation 62.126) can be replaced by the test | Pi41 — Pj | < ps, which is in terms of the value 
function. 


62.4 ADP-Based Optimal Controller Design: Examples 


We now present optimal controller design results that were obtained considering a power system with a 
linear model [3]. For the same system we will use both a discrete-time and a continuous-time approach to 
ADP to obtain the state-feedback controller, which optimizes a quadratic cost index. This is the familiar 
LQR problem. 

We will first present the results obtained using the continuous-time HDP approach. We then show 
that the same results will be obtained using the discrete-time ADHDP algorithm, when considering a very 
small sampling period. 


62.4.1 Power System and Cost Function 


Even though power systems are characterized by nonlinearities, linear state-feedback control is regularly 
employed for load-frequency control at a certain nominal operating points that are characterized by 
small variations of the system load around a constant value. Although this assumption seems to have 
simplified the design problem of a load-frequency controller, a new problem appears from the fact that 
the parameters of the actual plant are not precisely known. For this reason it is particularly advantageous 
to apply model free ADP methods to obtain the optimal LQR controller for a given operating point of the 
power system. 

The continuous-time linear model of the system that is considered here, [21], is characterized by the 
realistic values 


0.665 8 0 0 
au-| 0 73.663 3.663 0 
—6.86 0 13.736 —13.736 (62.127) 
6 0 0 0 


B'=[0 0 13.736 0] 


One can obtain the discrete version of this model by discretization using a zero-order hold method 
with the sample time T=0.01s. 
The infinite horizon cost function is defined for the continuous-time case as 


V= / [x? (2) Qx(t) + u? (t)Ru(t) dt (62.128) 
t 


where the matrices Q and R were chosen to be identity matrices of appropriate dimensions. For the case 
of the discrete-time ADP algorithm the cost function is 


[o,0) 
V=)0 (x/ Qaxi+ uj Raui) (62.129) 

i=k 
where the parameters, given by the matrices Qy and Rg, were chosen as Qx T and Rx T, where T 
denotes the sampling period. One sees that for the closed-loop system with state-feedback controller in 
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the form u = —Kx the two cost functions can be expressed as V(x(t)) = x! (t)Px(t) and V(x;) = uPhey 
respectively. 


62.4.2 Continuous-Time Value Iteration 


Online Value Iteration Algorithm 


Initialize: A restriction on the initial matrix Po such that the corresponding Ko = —R7!B' Py bea 
stabilizing controller is not required. Thus, we choose to start the ADP algorithm considering 
the case in which the system operates without controller (ie., Pop =Onx, and h (x) = Ko 
x = —R7!B" Pox). Let j = 0. 

Value Update Step: Solve for the parameters given by the matrix Pj, using 


t+T 
x! (t)Pj41x(t) = | (x? Qu + (hY (x))? RA (x))dt + x" (t + T)Pjx(t + T). (62.130) 
t 


The value function update amounts to the update of the kernel matrix Pj. 
Policy Update Step: Determine an improved policy using 


AU+Y (x) = —RoBT Pi x = Kjyix (62.131) 
Stop Algorithm Step: If 
|| Pita — Pjl| < eps (62.132) 


then STOP, else let j = j + 1 and go to the value update step. 


To implement this iteration scheme, one only needs to know the value of the matrix, which is 
required explicitly in the policy update step. The information on is not required at any step of the 
algorithm. The states, x(t) and x(t + T), and the reward over the time interval [t, f+ T], d(t, T, h(.)) = 

ae r(x(t), h(x(t)))dt, are observed on-line. Representing the value function as V(x) =x! Px = 
(vec(P))" (x @ x) = pix then the value update step can be written as 


Pirx(t) = d(t, T,A'(.)) +p) X(t + T) (62.133) 


Using this equation one can now solve for the parameters p;+1, in the least-squares sense, after collecting 
online enough state-trajectory points defined by the triplets (x(t), x(t + T), d(t, T, h')(.))) to set up a 
solvable problem. The target will be the right-hand side of Equation 62.133 and the least-squares solution 
will provide the parameters of the matrix Pj+1. 

Since the symmetric matrix P; has a number of 10 different parameters, each iteration step requires 
collecting at least 10 state-trajectory points in order to set up a solvable least-squares problem. We will 
choose to solve for the parameters of P; using a batch least-squares method considering 15 data points 
collected along the state trajectory of the system. This is because we intend to compare the results obtained 
with the continuous-time HDP algorithm with the ones resulting from the application of the discrete- 
time Q-learning algorithm (i.e. ADHDP strategy) and the Q-function in this case has a number of 15 
parameters (which require the measurement of 15 state points in order to solve a least-squares problem). 

For the implementation of the continuous-time value iteration algorithm the state measurements were 
taken at each 0.1s time period, such that a cost function update was performed at each 1.5 s. Over 60 s a 
number of 40 iterations of value function and control policy updates were performed. The convergence 
of some of the Critic parameters, namely P(1, 1), P(1, 3), P(2, 4), and P(4, 4), is presented in Figure 62.3. 
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FIGURE 62.3 Convergence of the P matrix parameters obtained online using the continuous-time HDP algorithm. 


The optimal control solution obtained after running the algorithm (without using knowledge regarding 


the system matrix A) is 


0.4753 0.4771 0.0602 0.4770 
0.4771 0.7838 0.1238 0.3852 
0.0602 0.1238 0.0513 0.0302 
0.4770 0.3852 0.0302 2.3462 


Pcr—Hpp = 


while the exact solution of the continuous-time algebraic Riccati equation is 


0.4750 0.4766 0.0601 0.4751 
0.4766 0.7831 0.1237 0.3829 
0.0601 0.1237 0.0513 0.0298 
0.4751 0.3829 0.0298 2.3370 


PCARE = 


(62.134) 


(62.135) 


Comparing the values of the two matrices given by Equations 62.134 and 62.135 one sees that the 
solution obtained online using the ADP algorithm and without using any knowledge on the internal 
dynamics of the system specified by matrix A is very close to the exact solution obtained by solving the 


algebraic Riccati equation associated with the infinite horizon optimal control problem. 


62.4.3 Discrete-Time Action-Dependent Value Iteration 


Discrete-Time Action-Dependent Value Function-Q-Function 


The Q-function for the LQR case is defined as 
Q* (xk, UR) = x¢ Qaxz + ug Rag + Rei Mas 


where x,41 = Ax; + Buz, and P* denotes the parameters of the optimal value function. 
With the notation ra = Lee ug] the Q-function will have the form 


T 
Q* (xz; Uk) — z) Hz = | fe td | 


Uk A, ux A, uu Uk 


where Hy, = Qg +A! P*A, Hy, = A! P*B, Hy, = B' P*A, and Hy, = Rg + B'P*B. 


(62.136) 


(62.137) 
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The optimal Q-function is equal with the optimal value function when u, is the optimal policy 
eo) — min OF ee 2 oT T 4 T)F 
V" (xp) = _ Q* (xk, UK) = min Es up | H [seg uf | ! (62.138) 


Given the optimal Q-function then the optimal control policy is obtained from 


OQ* (xz, u 
Oe 4k) _ 9 (62.139) 
Our 
which implies that 2Hyxx% + 2Hyy,u, = 0, thus 
ut = Hy) Huxxk- (62.140) 


The second-order sufficiency condition for the minimization is 


O° Q* (xKs Ux) 


Die >0, that is Hy, > 0, (62.141) 
k 


and it is satisfied since H,,,, = Ry + B' P*B, with Ry > 0 and P* > 0. 
In terms of the optimal Q-function one has the following recurrence relation 


Q* (kes Ue) = x, Qaxe + up Rae + te Q* (Xk Mkt) (62.142) 
+1 


which is a fixed-point equation. Based on this equation, and using the representation of the optimal 
Q-function (Equation 62.137), the online Q-learning algorithm is now formulated. 


Online Q-Learning Algorithm 


Initialize: A restriction on the initial matrix Ho such that the corresponding ho(x;,) = H., as Oe = 
Kox, is a stabilizing controller is not required. Thus, we choose to start the ADP algorithm 
considering the case in which the system operates without controller (i.e., Ho = 0(n+m)x(n+m) 
and ho(xz) = Hy HuxoXk): Let j = 0. 

Value Update Step: Determine online, the parameters of the Q-function, given by the matrix Hj+1, 


such that the equation 
Ze Hj 1Zk = 1(Xhs UK) + 2  Ajzey» (62.143) 


where uz = hj(xx) = Kjxx and ug, = hj(xe41) = Kjx+41, is solved in the least-squares sense. 
The value function update amounts to the update of the kernel matrix Hj. 
Policy Update Step: Determine an improved policy using 


Iya (xk) = Aya py Hua 1xk = Kj 1xk- (62.144) 
Stop Algorithm Step: If 
(His — Hill < ex) (62.145) 
then STOP, else let j = j + 1 and go to the value update step. 


We now present details related to the online implementation of the value update step. 
Ina similar representation with the one given for the value function in Equation 62.26, the Q-function 
can be represented as 


Qlzr) = zp Hee = HTH (62.146) 


where H = vec(H) and Z = Zz ® Zk. 
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FIGURE 62.4 Convergence of the P matrix parameters for online discrete-time ADHDP algorithm. 


Then the value update step becomes 
Fy 12k = (xk Ue) + A Zk. (62.147) 
The vector of parameters of the Q-function Hj, is found by minimizing the error 
ek = 1(xK, MK) + Ay Zep — Ay Ze (62.148) 


in least-squares sense over the compact set 2. 

The relation (Equation 62.144) shows that the control signal u, is a linear combination of the state 
vector. Thus, not all the elements of the vector z;, are linearly independent. This fact causes the problem 
of insolvability of Equation 62.144. 

This problem is removed by adding exploration noise to the control signal. Thus, let ue = ug + nk 
where n;(0,0)is exploration noise with zero mean and variance o. With this modification the vector z, 
becomes 


Zhe = [xp tefl” = [xp (ug y)"I", (62.149) 


and now obtaining the online solution of Equation 62.147 is guaranteed by the excitation condition. 

In Figure 62.4 is presented the convergence of the critic parameters when the discrete-time ADHDP 
algorithm was used. The duration of the simulation was 60 s, during which a number of 400 cost function 
(Q-function) updates took place. Each update required 15 measurements of the state of the system, which 
were taken with sample time of T = 0.01 s, since there are 15 independent parameters of the Q-function, 
given by the elements of the symmetric matrix H. 

The optimal value function can be calculated based on the obtained optimal Q-function and the 
obtained optimal control policy using the equation 


P* =[I, K*"]H* A (62.150) 
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In our example the solution for the parameters of the optimal value function obtained after running 
the algorithm is given in the matrix 


0.4802 0.4768 0.0603 0.4754 
0.4768 0.7887 0.1239 0.3834 
Poe - 2.151 
DT—-ADHDP ~ | 9.0603 0.1239 0.0567 0.0300 ede) 


0.4754 0.3843 0.0300 2.3433 


Comparing the results presented in Figure 62.4 with the ones given in Figure 62.3 one sees that 
the discrete-time online algorithm converges to the optimal solution of the continuous-time algebraic 
Riccati equation for this case in which a very small sampling period was considered. It is clear that 
fairly equal amounts of time are required for both the continuous-time and the discrete-time ADP 
algorithms to converge, however, the discrete-time ADHDP algorithm is computationally more intensive 
(400 required iterations) than the continuous-time HDP (40 iterations). The advantage of using the 
discrete-time algorithm consists in its model-free characteristic while the presented continuous-time 
algorithm still requires knowledge of the system’s B matrix. This observation motivates further research 
for finding a continuous-time model-free ADP approach to solve the optimal control problem. 


Further Reading 


For those interested in applying ADP methods and/or developing new ADP algorithms, we suggest for 
further reading the recently published papers [4,22], and the papers referred therein. The consistent lists 
of references are accurately encompassing the numerous results in this fast growing research field. 
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63.1 Introduction 


In many applications where dynamical system models are used to describe the behavior of a real world 
system, stochastic components and random noises are included in the model to capture uncertainties in 
the operating environment and the system structure of the physical process being studied. The analysis 
and control of such systems then involves evaluating the stability properties of a random dynamical 
system. Stability is an important property of a system and we know from classical control theory that 
input-output stability is a necessary condition for control system design, but are aware that analytic 
techniques for evaluating stability are often restricted to linear systems, or special classes of nonlinear 
systems. The general study of the stability properties of stochastic dynamical systems is important and 
considerable effort has been devoted to the study of stochastic stability. Significant results have been 
reported in the literature with applications to physical and engineering systems. 

A comprehensive survey on the topic of stochastic stability was given by [27], and since that time there 
have been many significant developments of the theory and its applications in science and engineering. 
In this chapter, we present some basic results on the study of stability of stochastic systems. Because of 
limited space, only selected topics are presented and discussed. We begin with a discussion of the basic 
definitions of stochastic stability and the relationships among them. Kozin’s survey provides an excellent 
introduction to the subject and a good explanation of how the various notions of stochastic stability are 
related. 

It is not necessary that readers have an extensive background in stochastic processes or other related 
mathematical topics. In this chapter, the results and methods will be stated as simply as possible and 
no proofs of the theorems are given. When necessary, important mathematical concepts that are the 
foundation to some of the results, will be discussed briefly. This will provide a better appreciation of the 
results, the application of the results, and the key steps required to develop the theory further. Those 
readers interested in a particular topic or result discussed in this chapter are encouraged to go to the 
original papers and the references therein for more detailed information. 
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There are at least three times as many definitions for the stability of stochastic systems as there are for 
deterministic systems. This is because in a stochastic setting there are three basic types of convergence: 
convergence in probability, convergence in mean (or moment), and convergence in an almost sure 
(sample path, probability one) sense. Kozin [27] presented several definitions for stochastic stability, and 
these definitions have been extended in subsequent research works. Readers are cautioned to examine 
carefully the definition of stochastic stability that is being used when interpreting any stochastic stability 
results. 

We begin with some preliminary definitions of stability concepts for a deterministic system. Let 
x(t; x, to) denote the trajectory of a dynamic system initial from xo at time fo. 


Definition 63.1: Lyapunov Stability 


The equilibrium solution, assumed to be 0 unless stated otherwise, is said to be stable if, given ¢ > 0, 
8(8, to) > 0 exists so that, for all ||xo|| < 8, 


sup [lx(t5 x05 fo)ll <€. 
t>to 


Definition 63.2: Asymptotic Lyapunov Stability 


The equilibrium solution is said to be asymptotically stable if it is stable and if 8’ > 0 exists so that 
\|xo|l < 8’, guarantees that 


lim ||x(t; x9; to) || = 0. 
t—co 


If the convergence holds for all initial times, to, it is referred to as uniform asymptotic stability. 


Definition 63.3: Exponential Lyapunov Stability 


The equilibrium solution is said to be exponentially stable if it is asymptotically stable and if there exists a 
8 >0,ana> 0, andaB > 0 so that ||xo|| < 8 guarantees that 


\|x(ts xoto) || < Bllxoll exp (tt) 


If the convergence holds for all initial times, to, it is referred to as uniform exponential stability. 


These deterministic stability definitions can be translated into a stochastic setting by properly inter- 
preting the notion of convergence, that is, in probability, in moment, or almost surely. For example, in 
Definition 63.1 for Lyapunov stability, the variable of interest is sup; t) ||x(¢; xo, fo) |], and we have to study 
the various ways in which this (now random) variable can converge. We denote the fact that the variable 
is random by including the variable w, that is, x(t;xo, to, o), and we will make this more precise later. 
Then, 


Definition 63.4: I,: Lyapunov Stability in Probability 


The equilibrium solution is said to be stable in probability if, givens, ¢' > 0, 8(€, &’, to) > 0 exists so that for 
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all \|xo|| < 8, 
P{sup ||x(t;xo, to, ©)|| > &} <e. 


t> to 


Here, P denotes probability. 


Definition 63.5: Ly: Lyapunov Stability in the pth Moment 


The equilibrium solution is said to be stable in pth moment, p > 0 if, given € > 0, 8(¢, to) > 0 exists so that 
\|xo|| < 8 guarantees that 


E{sup ||x(t;xo, to, o) ||P} <«. 


t>to 


Here, E denotes expectation. 


Definition 63.6: I,.s.: Almost Sure Lyapunov Stability 


The equilibrium solution is said to be almost surely stable if 


P{ lim sup ||x(t; xo, to, @)|| = 0} = 1. 


lxoll>0 ¢>t 


Note, almost sure stability is equivalent to saying that, with probability one, all sample solutions are 
Lyapunov stable. 


Similar statements can be made for asymptotic stability and for exponential stability. These definitions 
are introduced next for completeness and because they are often used in applications. 


Definition 63.7: II): Asymptotic Lyapunov Stability in Probability 


The equilibrium solution is said to be asymptotically stable in probability if it is stable in probability and if 
8’ > 0 exists so that ||xo|| < 8’ guarantees that 


lim P{sup ||x(t; x0, to, ©)|| > ¢} = 0. 
$00 t>8 


If the convergence holds for all initial times, fo, it is referred to as uniform asymptotic stability in 


probability. 


Definition 63.8: II,,: Asymptotic Lyapunov Stability in the pth Moment 


The equilibrium solution is said to be asymptotically pth moment stable if it is stable in the pth moment and 
if 8’ > 0 exists so that ||xo|| < 8’ guarantees that 


lim E{sup ||x(t;xo, to, )||} = 0. 
800 p58 
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Definition 63.9: II,,5.: Almost Sure Asymptotic Lyapunov Stability 


The equilibrium solution is said to be almost surely asymptotically stable if it is almost surely stable and if 
8’ > 0 exists so that ||xo|| < 8’ guarantees that, for any > 0, 


lim {sup ||x(t; xo, to, ©)|| > ¢} = 0. 


—> CO t>8 


Weaker versions of the stability definitions are common in the stochastic stability literature. In these 
versions the stochastic stability properties of the system are given in terms of particular instants, f, rather 
than the seminfinite time interval [fp, oo) as given in the majority of the definitions given above. Most 
noteworthy, are the concepts of the pth moment and almost sure exponential stability: 


Definition 63.10: III: pth Moment Exponential Lyapunov Stability 


The equilibrium solution is said to be pth moment exponentially stable if there existsa 8 > 0, ana > 0, and 
aB > 0so that ||xo|| < 8 guarantees that 


E{||x(t3 xo, to, ©) ||} < Bllxoll exp "°¢- . 


Definition 63.11: III,,,,: Almost Sure Exponential Lyapunov Stability 


The equilibrium solution is said to be almost surely exponentially stable if there exista8 >0,ana> 0, 
and aB > 0 so that ||xo|| < 8 guarantees that 


P{{||x(t; x0 fos ©) II} < Bllxoll exp" *—} = 1 


Example 63.1: 
Consider the scalar Ito equation 
dx(t) = ax(t)dt + ox(t) dwi(t) (63.1) 


where {w(t)}t>0 is a standard Wiener process. The infinitesimal generator for the system is given by 


2 


ese d 
L= 50x pea ae (63.2) 


The solution process x; for t > 0 is given by 
x(t) = (4-30 It oo Sp dws) x0. 


Hence, 


t 

x(t) tt es 

log i =(a soto f aw, 
0 
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and the asymptotic exponential growth rate of the process is 


t 
big ck x(t) 1 , _ 6 1 
eon = 108 a =(a ao pee dw(s)s=a— 0 


2 


0 


The last equality follows from the fact that the Wiener process, with increment dw(t), is a zero-mean 
ergodic process. We then conclude that the system is almost surely exponentially stable in the sense that 


Pxo { Jim, x(t) = 0, at an exponential rate as.} =1, 


if, and only if, a < 50°. 

Next we compare this with the second-moment stability result; see also Example 63.2 in the next section 
where the same conclusion follows from a Lyapunov stability analysis of the system. From our previous 
calculation, x? for t > 0 is given by 

Re = e2la—70")t p20 7 Bins 
Then, 
2. 
E{x?} = e@2to Vt E(x2}, 


and we conclude that the system is exponentially second-moment stable if, and only if, (a+ 507) <0, 
ora <— 507. Therefore, unlike deterministic systems, even though moment stability implies almost sure 
(sample path) stability, almost sure stability need not imply moment stability of the system. For the system 
Equation 63.1, the pth moment is exponentially stable if and only if a < 507(1 — p) where p = 1,2,3,.... 

In the early stages of the development of stability criteria for stochastic systems, investigators were 
primarily concerned with moment stability and stability in probability; the mathematical theory for the 
study of almost sure (sample path) stability was not yet fully developed. During this initial development 
period there was some confusion about the basic stability concepts, their usefulness in applications, and 
the relationship among the different stability concepts. Kozin’s survey clarified some of the confusions 
and provided a good foundation for further work. During the past 25 years, almost sure (sample path) 
stability studies have attracted increasing attention of researchers. This is not surprising because it is the 
sample paths rather than moments that are observed in real systems, and the stability properties of the 
sample paths can be most closely related to their deterministic counterpart, as argued by Kozin [27]. 
Practically speaking, moment stability criteria, when used to infer sample path stability, are often too 
conservative to be useful in applications. 

One of the most fruitful and important advances in the study of stochastic stability is the development of 
the theory of Lyapunov exponents for stochastic systems. This is the stochastic counterpart of the notion 
of characteristic exponents introduced in Lyapunov’s original work on asymptotic (exponential) stability. 
This approach provides necessary and sufficient conditions for almost sure asymptotic (exponential) 
stability, but significant computational problems must be solved. The Lyapunov exponent method uses 
sophisticated tools from stochastic process theory and other related branches of mathematics, and has the 
potential of providing testable conditions for almost sure asymptotic (exponential) stability for stochastic 
systems. We will provide an introduction to this approach for analyzing the stability of stochastic systems. 

In this chapter, we divide the results into two categories, the Lyapunov function method and the 
Lyapunov exponent method. 


63.2 Lyapunov Function Method 


The Lyapunov function method, known as Lyapunov’s second (direct) method, provides a powerful tool 
for the study of stability properties of dynamic systems because the technique does not require solving the 
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system equations explicitly. For deterministic systems, this method can be interpreted briefly as described 
in the following paragraph. 

Consider a nonnegative continuous function V(x) on R” with V(0)=0 and V(x) >0 for x #0. 
Suppose for some m € R, the set Qy = {x € R" : V(x) < m} is bounded and V(x) has continuous first 
partial derivatives in Q,,. Let the initial time to = 0 and let x(t) = x(t, xo) be the unique solution of the 
initial value problem: 

x(t)=f(x(t)), £20, 
fe =x ER", f(0)=0, Ga) 


for x9 € Qn. Because V(x) is continuous, the open set Q, for r € (0, m] defined by Q, = {x € R”: V(x) < 
r}, contains the origin and monotonically decreases to the singleton set {0} as r > 07. The total derivative 
V(x) of V(x) (along the solution trajectory x(t, xo)) is given by 


dV(x 
dt 


) oof OV def 
=f" (x): aie k(x). (63.4) 


V(x) = 


If —k(x) <0 for all x € Qm, with k(x) continuous, then V(x;) is a nonincreasing function of t, and 
V(xo) < m implies V(x(t)) < m for all t > 0. Equivalently, x9 € Qm implies that x; € Qm, for all t > 0. 
This establishes the stability of the zero solution of Equation 63.3 in the sense of Lyapunov, and V(x) is 
called a Lyapunov function for Equation 63.3. Let us further assume that k(x) > 0 for x € Q,,\{0}. Then 
V(x(t)), as a function of ft, is strictly monotone decreasing. In this case, V(x(t)) > 0 as t > +00 from 
Equation 63.4. This implies that x(t) > 0 as t > +-oo. This fact can also be seen through an integration 
of Equation 63.4, that is, 


t 
0 < V(xo) — V(x(t)) = / k(x;)ds<+0oo fort €[0,+00). (63.5) 
0 


It is evident from Equation 63.5 that x(t) > {0} = {x € Qm : k(x) = 0} as t > +00. This establishes the 
asymptotic stability of the trajectories of system Equation 63.3. 

The Lyapunov function V(x) can be interpreted as a generalized energy function of the system Equa- 
tion 63.3, and the above argument illustrates the physical intuition that if the energy of a physical system 
is always decreasing near an equilibrium state, then the equilibrium state is stable. 

Since Lyapunov’s original work, this direct method for stability study has been extensively investigated. 
The main advantage of the method is that one can obtain considerable information about the stability 
of a given system without explicitly solving the system equation. One major drawback of this method is 
that for general classes of nonlinear systems a systematic method for constructing a suitable Lyapunov 
function does not exist, and stability criteria (usually sufficient conditions) determined using the method 
depend critically on the Lyapunov function that is chosen. 

The first attempts to generalize the Lyapunov function method to stochastic stability studies were 
made by [8] and [24]. A systematic treatment of this topic was later given by [22,30,31] (primarily for 
white-noise stochastic systems). The key idea of the Lyapunov function approach for a stochastic system 
is the following: 

Consider the stochastic system defined on a probability space (Q, F, P), where Q is the set of elementary 
events (sample space), F is the o field that consists of all measurable subsets (events) of 2, and P is a 
probability measure: 

Ce =f(x(t),), t>0, (63.6) 


x(0) = xo. 


It is not reasonable to require that V(x(t)) <0 for all w, where x(t) = x(t, x9, @) is a sample solution of 
Equation 63.6 initial from xo. Note, when t is fixed, x(t, xo, ) is a random variable on (Q,F, P), and 
when w is fixed (a = w*), x(t, x9, o*) is a deterministic time function. The basic idea to insure “stability” 
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of the system is that the time derivative of the expectation of V(x(t)), denoted by £V(x(t)), is nonpositive. 
Here, £ is the infinitesimal generator of the process x(t). Suppose that the system is Markovian so that 
the solution process is a strong, time homogeneous Markov process. Then CL is defined by 


Exy(V(xat)) — V(x) 


a (63.7) 


eVtc)= Ji, 


where the domain of £ is defined as the space of functions V(x) for which Equation 63.7 is well- 
defined. This is a natural analog of the total derivative of V(x) along the solution trajectory x(t) in the 
deterministic case. Now suppose that, for a Lyapunov function V(x) that satisfies the conditions stated 
above, LV (x) < —k(x) < 0. It follows that 


t t 


0 < V(x0) — Exp V(x(t)) = Exy / k(x(s)) ds = —Ex, i LV(x(s)) ds < +00 (63.8) 
0 0 
and for t,s > 0 
Ex) V(x(t + s)) — V(x(s)) <0 as. (63.9) 


Equation 63.9 means that V(x(f)) is a supermartingale, and, by the martingale convergence theorem, we 
expect that V(x(t)) > 0 a.s. (almost surely) as t > +00. This means that x(t) > 0 as. as tf— +00. A 
similar argument can be obtained by using Equation 63.8 an analog of Equation 63.5. It is then reasonable 
to expect from Equation 63.8 that x(t) > {x € R" : k(x) = 0} almost surely. These are the key ideas behind 
the Lyapunov function approach to the stability analysis of stochastic systems. 

Kushner [30-32] used the properties of strong Markov processes and the martingale convergence the- 
orem to study the Lyapunov function method for stochastic systems with solution processes which are 
strong Markov processes with right continuous sample paths. A number of stability theorems were devel- 
oped in these works and the references therein. Kushner also presented various definitions of stochastic 
stability. The reader should note that his definition of stability “with probability one” is equivalent to 
the concept of stability in probability introduced earlier here. Also, asymptotic stability “with probability 
one” means stability “with probability one” and sample path stability, that is, x(t) > 0 a.s. as t > +oo. 
The key results of Kushner are based on the following supermartingale inequality that follows directly 
from Equation 63.8 where xo is given: 

V(x0) 


Py{ sup V(x(t))>e}< ; (63.10) 
0<t<+o0 € 


From this, the following typical results were obtained by Kushner. For simplicity, we assume that x(t) € 
Qm almost surely for some m > 0 and state these results in a simpler way. 


Theorem 63.1: Kushner 


1. Stability “with probability one”: 
If LV(x) <0, V(x) > 0 for x € Qn \{0}, then the origin is stable “with probability one”. 
2. Asymptotic stability “with probability one”: 
If LV (x) = —k(x) < 0 with k(x) > 0 for x € Qm\{0} and k(0) = 0, and if for any d > 0 small, ¢g > 0 
exists so that k(x) > d for x € {Qm : ||x|| = €q}, then the origin is stable “with probability one” and 
V(xo) 


Py {x(t) > 0 ast—> +oo} >1- . 
m 


In particular, if the conditions are satisfied for arbitrarily large m, then the origin is asymptotically 
stable “with probability one.” 
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3. Exponential asymptotic stability “with probability one”: 
If V(x) > 0, V(0) = 0 and LV(x) < —aV(x) on Qm for some a > 0, then the origin is stable “with 
probability one,” and 


V (x0) 2 uae 


Py | sup V(x(t)) > | < , VWI>0. 


T<t<+0oo 


In particular, if the conditions are satisfied for arbitrarily large m, then the origin is asymptotically 
stable “with probability one,” and 


Px, sup. V(x(t)) > 


—aT 
< Vi(xo)e 
T<t<+00 n 


Many interesting examples were also developed in Kushner’s work to demonstrate the application 
of the stability theorems. These examples also illustrated some construction procedures for Lyapunov 
functions for typical systems. 


Example 63.2: Kushner 


Consider the scalar Ito equation 


dx(t) = ax(t) dt + ox(t) dw(t) (63.11) 
where w(t) is a standard Wiener process. The infinitesimal generator for the system is given by 
1 d* d 
L= —o2x? —_ +.ax—. (63.12) 


2 dx2 dx 
If the Lyapunov function candidate is V(x) = x2, then 
LV(x) = (0? + 2a)x?. (63.13) 


If o2 + 2a < 0, then with Qn = {x: xr< m}, from (1) of the previous theorem, the zero solution is 
stable “with probability one.” Let m — +00. By (2) of the theorem, 


lim x(t) =0 a.s. (63.14) 
t>o 
where x(t) is the solution process of Equation 63.11. By (3) of the theorem, 
V(xg)e7 oF 
Prof sup x2=n}< ole (63.15) 
T<t<+00 » 


for some a > 0. 


Remark 63.1 


As calculated in the previous section, the asymptotic exponential growth rate of the process is 


t 

1, _ oO 1, 

(a a )+ tim © f dws) =a So? 
0 


We conclude that the system is almost surely exponentially stable in the sense that 
Py, | lim x(t) = 0, at an exponential rate a.s. | =1, 
t>oo 


if, and only if\a < 507. Compare this with the stability result a < — 50, given above, using the Lyapunov 
function method. Note that a < —4o? is actually the stability criterion for second-moment stability, 
which is a conservative estimate of the almost sure stability condition a < 507. 
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Has’minskii [22] and the references cited therein, provide a comprehensive study of the stability of 
diffusion processes interpreted as the solution process of a stochastic system governed by an Ito stochastic 
differential equation of the form, 


k 
dx(t) = b(t, x) dt + > o,(t,x) dt,(t), t>s (63,16) 


x(s) = Xs, 


where ¢,(t) are independent standard Wiener processes and the coefficients b(t, x) and o;,(t, x) satisfy 
Lipschitz and growth conditions. In this case, the infinitesimal generator £ of the system (associated 
with the solution process) is a second-order partial differential operator on functions V(t, x) that are 
twice continuously differentiable with respect to x and continuously differentiable with respect to t. £ is 
given by 


OV. se DV? Le av 
LV(t,x) = ane S bi(t, ae os X ail) aoe (63.17) 

The key idea of Has’minskii’s approach is to establish an inequality like Equation 63.10 developed in 
Kushner’s work. Below are some typical results obtained by Has’minskii; the reader is referred to [22] for 
a more detailed development. 

Let U bea neighborhood of 0 and U; = {t > 0} x U. The collection of functions V(t, x) defined in Uj, 
that are twice continuously differentiable in x except at the point x = 0 and continuously differentiable 
in ft, are denoted by C3(U;). A function V(t, x) is said to be positive definite in the Lyapunov sense if 
V(t, 0) = 0 for all t > 0 and V(t, x) => w(x) > 0 for x £0 and some continuous function w(x). 


Theorem 63.2: Has’minskii 


1. The trivial solution of Equation 63.16 is stable in probability (same as our definition) if there exists 
V(t,x)€ Clu; ), positive definite in the Lyapunov sense, so that LV(t,x) <0, forx #0. 

2. If the system Equation 63.16 is time homogeneous, that is, b(t, x) = b(x) and o,(t, x) = 0,(x) and if 
the nondegeneracy condition, 


3 ay(x)ridj > m(x) > MPs forkh=(d1...dn) ER", (63.18) 


ij=1 


is satisfied with continuous m(x) > 0 forx 4 0, then a necessary and sufficient condition for the trivial 
solution to be stable in probability is that a twice continuously differentiable function V(x) exists, 
except perhaps at x = 0, so that 


O-V 
LoV(x) = Phe = +a) FV <9 
: 


where Lo is the infinitesimal generator of the time homogeneous system. 
3. Ifthe system Equation 63.16 is linear, that is, b(t, x) = b(t)x and o,(t, x) = 0;,(t)x, then the system is 
exponentially p-stable (the pth moment is exponentially stable), that is, 


Exo {||x(ts x0 S)||?} < A: Ilx|? exp{—a(t a s)}, Pp > 0, 


for some constant a > 0 if, and only if, a function V(t, x) exists, homogeneous of degree p in x, so that 
for some constants kj > 0, i= 1,2, 3,4, 


ky |lxll? < V(t.x) < kollel’, LV (t,x) < —ksllxll?, 
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and 


OV o*V 
—||<k Pare ——||<k p-2. 
I Dx ll < Kall) I ae ll < Kall) 


In addition to the stability theorems, Has’minskii also studied other asymptotic properties of stochastic 
systems and presented many interesting examples that illustrate the stabilizing and destabilizing effects 
of random noise in stochastic systems. 

Just as in the case of deterministic systems, the Lyapunov function approach has the advantage that 
one may obtain considerable information about the qualitative (asymptotic) behavior of trajectories of 
the system, in particular stability properties of the system that are of interest, without solving the system 
equation. However, no general systematic procedure exists to construct a candidate Lyapunov function. 
So even though the theorems, like Has’minskii’s, provide necessary and sufficient conditions for stability, 
one may never find the “appropriate” Lyapunov function in a given application to determine the stability 
(or instability) of the system. Further, the nature of the stability condition obtained using the Lyapunov 
method depends critically on the choice of Lyapunov function that is used. Various techniques have 
been proposed by investigators to construct a suitable family of Lyapunov functions to obtain the “best” 
stability results. In the following, we summarize some of these efforts. 

There are several works related to the stability of linear systems of the form, 


He =[A+F(t)]x(t); t>0, (63.19) 


x(0) = x9 € R", 


with A a stable (Hurwitz) matrix and F(t) a stationary and ergodic matrix-valued random process. This 
problem was first considered by Kozin [26] by using the Gronwall-Bellman inequality rather than a 
Lyapunov function technique. Kozin’s results were found to be too conservative. Caughey and Gray [12] 
were able to obtain better results by using a very special type of quadratic form Lyapunov function. Later, 
Infante [23] extended these stability theorems by using the extremal property of the so-called regular 
pencil of a quadratic form. The basic idea behind Infante’s work is the following: 

Consider the quadratic form Lyapunov function V(x) = x’Px. Then the time derivative along the 
sample paths of the system Equation 63.19 is 


V (x(t) = x(t)! (F(t)P + PF(t))x(t) — x(t)’ Qx(t) (63.20) 


where Q is any positive-definite matrix and P is the unique solution of the Lyapunov equation 


A'P+PA=-Q. (63.21) 
If X(t) = V(x(t))/V(x(t)), then 
t 
V(x(t)) = V(xo) expt | X(s) ds}. (63.22) 
0 


From the extremal properties of matrix pencils, 
A(t) < Amaxl(A + F(t))’ + Q(A + F(t))Q"]. (63.23) 
Here, \max(K) denotes the largest magnitude of the eigenvalues of the matrix K. By the ergodicity property 
of the random matrix process F(t), we have the almost sure stability condition 
t 
fim © [ x) de = EIA} S Emagl(A+ POY + QA + FOG <0. 


0 


Man [39] tried to generalize the results of Infante. However, several obvious mistakes can be observed 
in the derivation of the theorem and the two examples given in this work. 
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Following Infante et al. [29] used distributional properties of the random coefficient matrix F(t) to 
obtain improved results for two specific second-order systems in the form, 


ee + 2Bx(t) + [e+ f(H]x(H) = 0, (63.24) 


x(t) + [2B + g(t)]x(t) + cx(t) = 0, 


where f(t) and g(t) are stationary and ergodic random processes. 

Parthasarthy and Evan-Zwanoskii [48] presented an effective computational procedure, using an opti- 
mization technique, to apply the Lyapunov type procedure to higher order systems in the form of Equa- 
tion 63.19 with F(t) = k(t)G. Here G is a constant matrix and k(t) is a scalar (real-valued) stationary and 
ergodic random process. After proper parameterization of the quadratic form Lyapunov function, the 
Fletcher-Powell-Davidson optimization algorithm was used to optimize the stability region that depends 
on the system data and k(t). A fourth-order system was studied using the suggested procedure, and vari- 
ous simulation results were used to show that this procedure yielded a stability region that was not unduly 
conservative. Because an optimization procedure was used and the solution of a Lyapunov equation was 
required, this procedure required an extensive computational effort. 

Wiens and Sinha [56] proposed a more direct method for higher order systems defined as an intercon- 
nection of a set of second-order subsystems. Consider the system, 


Mx(t) + [Co + C(#)]x(t) + [Ko + K(t)]x(t) = 0, (63.25) 


where M, Co, and Ko are nonsingular n x n matrices and C(t), K(t) are n x n stationary and ergodic 
matrix-valued, random processes. The technique for constructing a Lyapunov function suggested by 
Walker [55] for deterministic systems was used to construct a quadratic form Lyapunov function V(x) 
for the deterministic counterpart of Equation 63.25, 


x1 0 I x1 ~a [*1 2nx2n 
= _ = ER 5 63.26 
(::) ee ete!) (”) ‘ Co ce 


that is 
1 
~ ~) : 5P3 ~ ~\ A xp / : 
V(x) =x 1 x; V(x)=x Px, P;=P;>0, fori=1,2, 
—-P3 Po 


with a time derivative along sample paths, 
V(x) = X/Aox, 
where Ag is properly defined and the following choices are made: 
P3=P,M~'Co, 
P} = P,M~'Ko+ 5(M~'Cay PMC), 


Then, Infante’s approach was used to obtain the following result: 


Theorem 63.3: Wiens and Sinha 


The system Equation 63.25 is almost surely asymptotically stable (Definition Is, in Section 63.1) in the 
large if a positive definite matrix P2 exists so that 
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1. P)M~!Kp is positive definite, 
2. The symmetric parts of P»M~!Cpo and (M~!Co)'P2(M~'!Ko) are positive definite, and 
3. E{X max[(Ay + C(t) + K(t))P~!]} < 0, 


where C(t) and K(t) are given by 


0 1 M-C9)'P»M— C(t) 
CHS 2 
5(M'Co)P2M C(t) {P>M~1(H)} + {P2M~!C(t)}’ 


and 


1 1 
K(t)- =| {MP Co)! PoM'K(1)} + 5(M Co) PM IK (1)! {P2M~1K(1)}" | 
P,M~'K(t) 0 
When applying their results to the second-order system Equation 63.24, Wiens and Sinha [56] obtained 
the stability criteria, 


E{f?(t)} <4B’c, and (63.27) 
E{g?(t)} < 4B7c/(c + 26%). (63.28) 


These results are similar to those obtained by Caughey and Gray [12]. Equation 63.27 is the “opti- 
mal” result of Infante, but Equation 63.28 is not. This result is not surprising, because no optimization 
procedure was used in deriving the stability criteria. The usefulness of the Wiens and Sinha theorem 
was demonstrated by applying their method to higher-order systems (n = 4 and 6 ) that yielded stability 
regions of practical significance. 

Another research direction for application of the Lyapunov function method is the study of stochastic 
feedback systems where the forward path is a time-invariant linear system and the random noise appears 
in the feedback path as a multiplicative feedback gain. In this work, Lyapunov functions are constructed 
by analogous methods for deterministic systems, for example, the Lyapunov equation, the path-integral 
technique and the Kalman-Yacubovich-Popov method. However, most of the results obtained can be 
derived by directly using a quadratic form Lyapunov function together with the associated Lyapunov 
equation. 

Kleinman [25] considered a stochastic system in the form, 


dx(t) = Ax(t) dt + Bx(t) d¢(t), (63.29) 


where ¢(f) is a scalar Wiener process with E{[¢(t) — ¢(t)]?} = o” | tt | . By using the moment equation 
and a quadratic Lyapunov function, the following necessary and sufficient condition for 0 to be “stable 
with probability one” (this is the same as Kushner’s definition) was derived: 


I@A+AQI+0°B@B 


is a stable matrix where “@” denotes the Kronecker product of matrices. However, this result should be 
carefully interpreted as discussed by Willems [58]. 
Willems [58,59] studied the feedback system in the form, 


(63.30) 


dx(t) = (Aox(t) — kbex(t)) dt — bex dc(t), 
x(0) = x9 € R", 


with ¢(t) a scalar-valued Wiener process with E{[¢(t) c(t) °} = 07 | t — tI, that is, a system consisting 
of a linear time-invariant plant in the forward path with rational transfer function H(s) and minimal 
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realization (Ag, b,c), and a multiplicative feedback gain that is the sum of a deterministic constant k 
and a stochastic noise component ¢(t). Suppose the system is written in companion form with state 
x(t) = (y(t), Dy(t),.-. D"—1y(t)) where D = d/dt and y(t) = cx(t) is the scalar output. The closed-loop 
transfer function is 


G(s) = c(sI — Ap — kbc) 1b = qs) (63.31) 


p(s) 
with p(s) and q(s) relatively prime because the realization is minimal. Then, the input-output relation 
can be written in the form, 


p(D) y(t) dt + q(D) y(t) dg(t) = 0. (63.32) 


Willems observed that if a positive-definite quadratic form V(x) was used as a Lyapunov function, then, 
following Kushner, £V(x) is the sum of a “deterministic part” obtained by setting the noise equal to 
zero and the term 507(q(D)y)?(@?V / Ox?) (%_, =D"! y). Here, £ is the infinitesimal generator of the 
system. To guarantee the negativity of £V(x) to assure stability, using Equation 63.32, Willems used a 
construction technique for a Lyapunov function that was originally developed by Brockett [10], that is 


t(x) 
V(x) = i [p(D)yh(D)y — (q(D)y)?] dt 
£(0) 


where the polynomial h(s) = s" + hy»-1s"-! +---+ po is the unique solution (assuming p(s) is strictly 
Hurwitz) of 


lisp s) + h(—)p(s)] = gls)q(—s) 


so that 
1 
LV (x) = —(q(D)y)” + 5 itn-19- (q(D)y). 


By applying Kushner’s results, Willems obtained the following theorem: 


Theorem 63.4: Willems 


1. The origin is mean-square stable in the large (in the sense that R(t) = E{x(t)x'(t)} <M < +00 for 


t > 0,x9 € R" and sup ||R(t)|| — 0 as ||R(O)|| + 0) and stable “with probability one” (in Kushner’s 
t>0 
sense) in the large, if p(s) is strictly Hurwitz and 


o7hy-1 


seh (63.33) 


Moreover, the following identity holds: 


o7hy—1 


ee) “2 +00 
ape’ 2. oes . 2 
7 = o [ison dt = aa i: | G(iw) |“ dw 
0 —0o 


where g(t) is the impulse response of the closed-loop (stable) deterministic system and g(t) and G(s) 
are Laplace transform pairs. 

2. If the inequality Equation 63.33 holds, the stability of the origin as indicated in condition 1 above is 
asymptotic. 
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Brockett and Willems [11] studied the linear feedback system given by 
x(t) = Ax(t) — BK(t)Cx(t) (63.34) 


where (A,B,C) is a completely symmetric realization, that is, AT =A € R™*", B=C’ ¢ R"™, and 
K(t) € R”*" is a stationary ergodic matrix process. By using a quadratic Lyapunov function, the specific 
properties of a completely symmetric system, and the well-known Kalman-Yacubovich-Popov Lemma, 
they obtained the following stability theorem: 


Theorem 63.5: Brockett and Willems 


For the system Equation 63.34, 
1. If K(t) = K’'(t) almost surely and 


max = E{Xmax(A a BK(t)C)} <0, 


then the origin is almost surely asymptotically stable (in the sense that | i x(t) = 0a.s.). In partic- 
—>+00 


ular, if m = 1, which is analogous to a single input-single output system, then 


= rs 1 dg(o)/do 
max: =~ d 
: | ( za) | go) |*° 


1 


where Z, is the largest zero of the open-loop transfer function g(s) = C(sI — A)~'B and p(-) is the 
density function of K(0). 
2. Ifm=1and 
qn-18" | ++++ +40 
s+ Dy-1s*@14---+ po’ 
the origin is almost surely asymptotically stable if a constant B exists so that 
a. E{min(, K(t))} > 0, 
b. The poles of g(s) lie in Re{s} < —qn-1f, 
c. The locus of G(iw — qn—18), —-CO < w < +00, does not encircle or intersect the closed disc 
centered at (—1/2B,0) with radius 1/26 in the complex plane. 


g(s) = C(sI — A) 1B = 


Mahalanabis and Purkayastha [38] as well as Socha [53], applied techniques similar to Willems and 
Willems and Brockett to study nonlinear stochastic systems. Readers should consult their papers and the 
references cited therein for more details on this approach. 

An extension of Kushner’s and Has’minskii’s work on the Lyapunov function approach uses the 
Lyapunov function technique to study large-scale stochastic systems. The development of large-scale 
system theory in the past several decades was an impetus for these studies. The Lyapunov function 
technique of Kushner and Has’minskii is based on the use of scalar, positive-definite functions and is 
not effective for large-scale systems. As in the deterministic case, the difficulties are often overcome, if a 
vector positive-definite function is used. Michel and Rasmussen [40,41,51] used vector valued positive- 
definite functions to study various stability properties of large-scale stochastic systems. Their approach 
was to construct a Lyapunov function for the complete system from those of the subsystems. Stability 
properties were studied by investigating the stability properties of the lower order subsystems and the 
interconnection structure. Ladde and Siljak in [33] and Siljak [52] established quadratic mean stability 
criteria by using a positive-definite vector Lyapunov function. This is an extension of the comparison 
principle developed by Ladde for deterministic systems to the stochastic case. In these works, a linear 
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comparison system was used. Bitsris [9] extended their work by considering a nonlinear comparison 
system. White noise systems were studied in all of the above works. Socha [54] investigated a real noise 
system where the noise satisfied the law of large numbers. Has’minskii’s result [22] was extended to a 
large-scale system in this work. Interested readers are referred to the original papers and references cited 
therein. 


63.3 The Lyapunov Exponent Method and the Stability 
of Linear Stochastic Systems 


One of the major advances in the study of stochastic stability during the past several decades is the 
application of the Lyapunov exponent concept to stochastic systems. This method uses sophisticated 
mathematical techniques to study the sample behavior of stochastic systems and often yields necessary 
and sufficient conditions for almost sure (sample) stability in the sense that 


lim ||x(t,xo,)||=0 as. (63.35) 
too 


Because this method focuses on the sample path behavior of a stochastic system, rather than the moments, 
it has the greatest potential for applications in science and engineering. In this section we present a 
summary of this method along with selected results. 

After the introduction of the concept of Lyapunov exponents by Lyapunov [37], this concept has 
formed the foundation for many investigations into the stability properties of deterministic dynamical 
systems. However, it is only recently that the Lyapunov exponent method has been used to study almost 
sure stability of stochastic systems. The setup is as follows: 

Consider the continuous time linear stochastic system defined by 


ea = A(t,w)x(t), t> 0, 


x(0) = xp € R4. (Zc) 


Let x(t, xo, w) denote the unique sample solution of (-) initial from xo for almost all w € Q. (We always 
denote the underlying probability space by (, F, P)). The Lyapunov exponent i,»(xq) determined by xo 
is defined by the random variable, 


= — 1 
Aeo(X0) = lims+ +005 log [lx(t, x0, o)Il, (63.36) 


for (X,). Although xo can be a random variable, for simplicity we will concern ourselves primarily with 
the case where xp is nonrandom and fixed. 

In the case A(t, w) = A(t) is a deterministic R¢*4-valued continuous bounded function, Lyapunov [37] 
proved the following fundamental result for the exponent (xo) for the system, 


x(t) = A(£)x(t) : (x) 
1. (xo) is finite for all xp € R4\{0}. 
2. The set of real numbers which are Lyapunov exponents for some xo € R“\{0} is finite with cardi- 
nality p, 1<p<d; 
—00 <1 < dg <+++<Ap<+too0, Ai Ee RVi. 
3. (exo) = (xo) for xo € R4\{0} and cE R\{0}. (axo + Byo) < max{i(xo), (yo)} for x0, vo € 
R4\{0} and a, 8 € R with equality if X(xo) < X(yo) and B £ 0. The sets £; = {x € R4\{0}: K(x) = 


hi}, i= 1,2,...,p, are linear subspaces of R4, and an ae is a filtration of R4, that is, 


{0} 2 Ly C Li C++» CLp =R" 


where d; = dim(£;) — dim(£j_ 1) is called the multiplicity of the exponent ; fori = 1,2,...,p and 
the collection {(Xj, di)yh_ , is referred to as the Lyapunov spectrum of the system (X). We have the 
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relation 


P 
1 — 1 
Do didg < Tims. 407 log 1 ®(E)|| < lime + +007 log | P() (63.37) 


i=1 


where ®(f) is the transition matrix of (2). The system is said to be (forward) regular if the two 
inequalities in Equation 63.37 are equalities. For a forward regular system the lim and lim can be 
replaced by lim. 


For the stochastic system (2,) with w € & fixed, the relationship given in Equation 63.36 implies that if 
Xeo(xo) <0, then the sample solution x(t, xo, w) will converge to zero at the exponential rate lhe (xo)| and, 
if Xe (xo) > 0, then the sample solution x(t, xo, w) cannot remain in any bounded region of R¢ indefinitely. 
From this we see that 4. (xo) contains information about the sample path stability of the system and as 
we will see later, in many cases a necessary and sufficient condition for sample path (almost sure) stability 
is obtained. 

Arnold and Wihstutz [5] have given a detailed survey of research work on Lyapunov exponents. The 
survey is mathematically oriented and presents a summary of general properties and results on the topic. 
Readers are encouraged to refer to Arnold and Wihstutz [5] for more details. Our focus, however, is more 
on the application of Lyapunov exponents to the study of stability of stochastic systems. 

The fundamental aspects of the Lyapunov exponent method were applied to stochastic systems by 
Furstenberg, Oseledec, and Has’minskii. We will briefly review the work of Oseledec and Furstenberg 
and then focus attention on the work of Has’minskii, that best illustrates the application of the Lyapunov 
exponent to the study of stochastic stability. 

The random variables (xo) given in Equation 63.36 are simple, nonrandom constants under certain 
conditions, for example, stationarity and ergodicity. This is a major consequence of the multiplicative 
ergodic theorem of Oseledec [46] that establishes conditions for the regularity of stochastic systems. In 
his original work, Lyapunov used the regularity of (X) to determine the stability of a perturbed version 
of the system (X) from the stability of (X). Although regularity is usually very difficult to verify for a 
particular system, Oseledec proved an almost sure statement about regularity. Because of our interest in 
the stability of stochastic systems, the theorem can be stated in the following special form for (,), see 
Arnold et al. [2]. 


Theorem 63.6: Multiplicative Ergodic Theorem: Oseledec 


Suppose A(t, w) is stationary with finite mean, that is, E{A(0, )} < oo. Then for (X,), we have 


1. State-space decomposition: 
For almost all w € Q, an integer r = r() exists with 1 < r(w) < d, real numbers \4(w) < h2(@) < 
+++ <d+(w), and linear subspaces (Oseledec spaces) E\(w),...,Er(w) with dimension dj(w) = 
dim[E;(w)] so that 


Ré= a E,(@) 
i=1 


and 


1 
_jim | ‘ log || P(t, w)xoll = Ai(w), if xo € E(w) 


where ®(t, w) is the transition matrix of (X_) and “Q)” denotes the direct sum of subspaces. 
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2. Domain of attraction of E;(): 
1 
jim. r log || P(t, o)xol] = dilw), iff xo € Li(w)\Li-1(), 


where Li(w) = Qj) Ej(o). 
3. Center of gravity of exponents: 
r(@) 1 
Y > di(w)di(m) = lim_ = log | det &(t, @)| = trE{Ao()|F} 
= t>+00 ft 
where F is the o algebra generated by the invariant sets of A(t, w). 
4. Invariance property: If A(t, w) is ergodic as well, then the random variables r(w), di(w), and d;(w), 
i=1,2,...,1r, are independent of w and are nonrandom constants. 


Note that under the current assumptions, all lim are actually lim and (3) is equivalent to the almost 
sure regularity of the stochastic system (X,). Oseledec’s theorem is a very general result and the above 
is a special version for the system (2). A detailed statement of the theorem is beyond the scope of this 
chapter. As far as sample path (almost sure) stability is concerned, the sign of the top exponent i, is of 
interest. We present a simple example to illustrate the application of the theorem. 


Example 63.3: 
Consider the randomly switched linear system 


1 1 
x(t) = mil — y(t))A_1x(t) + 5M +y(t)Aix(t), t>0, 
x(0) = Xo, 


(63.38) 


where 


-a 0 1 
Atel g ch =f A 0<b<a<+o00 0<c< +00, (63.39) 


and y(t) € {—1, +1} is the random telegraph process with mean time between jumps a—! > 0. Then, 


—at 
0 
Ait —|€ 63.40 
0 eW ot , ( ) 


ect tet 
rel cr): (63.41) 


The phase curves of these two linear systems are as shown in Figure 63.1. 


and 


It is easy to verify that, for xo € R?\{0}, 


as 1 1 k 
Ao(xo) = tim | rs log ||x(t, xo, ©) || = a+ lim: ; log evar ee vant xo ae (63.42) 
and 
- 1 1 
Xo (R2) 2 _lim , - log || det &(¢, 0)|| = 0 lim 5 log || det (cA. AW VI] as, (63.43) 
POO) 


where {y(k) : k > 1} is the embedded Markov chain of y(t) and {1 :k> lpisa sequence of i.i.d. random 
variables exponentially distributed with parameter a, and independent of {y(k) : k > 1}. Because y(t) is 
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FIGURE 63.1 Phase curves. 


stationary and ergodic with unique stationary (invariant) distribution P{y(t) = 1} = P{y(t) 1} 5» 


Oseledec’s theorem applies and a F-set No exists with P(No) = 0 and two real numbers ), and 2 (with 
possibly 4; = 2) so that 


Xeo(xo) € {A122} for xo € R?\{0} and we 2\No (63.44) 
From Equations 63.40, 63.41, and 63.43 and the law of large numbers, 


Ko(R2) = o- lim + log eet oe t--+e¥W) , eltant-+ta-ip), 
k>00 k 


1 . 1 
=—(a+b)o lim (1) +...4 1%) + 2cw lim =(Tay) +e + TKI)» 
k>00 k k>00 k 


—_ at oe (63.45) 
Cc 


Here i, is the number of times that y(j) takes the value —1 in k steps and {t, Tj); ij = 1} is a sequence 
of iid. random variables exponentially distributed with parameter a. 

Because E = span {e; = (1,0)!} isa common eigenspace of e4-"' and e“"', by the law of large numbers, 
it is clear that for any xp € E\{0}, 


= 1 1 
he (xo) = —32 + s as. for xo € E\{0}. (63.46) 


From (3) of the theorem, Equations 63.45 and 63.46, we obtain 


1 1 1 1 
hy ==c—=a and )23=-c——=b (63.47) 
2 2 2 2 


(with 47 < 2). We can identify the following objects in Oseledec’s theorem: 
r=2, E\(m)=E£, forme Q\N,, 


d}=1, E(w)=Et= span {e2 = (0, 1)"}, for w € Q\Ni, (63.48) 
dy=1, Li(w)=E, Lo(w)=R?, forme Q\N, 
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where Nj is a F-set with P(N;) = 0 and 


eee for xo € E\{0} and we Q\N, (63.49) 


RolXo) =2, forxy €R?\E and we Q\N,. 


From Equation 63.49 it follows that the system Equation 63.38 is almost surely exponentially stable in 
the sense that 
P{||x(£, @, x9) || > 0 as t > +00, at an exponential rate } = 1 


if and only if \2 < 0 (iffc < b). Note that the system is almost surely exponentially unstable if and only if 
1 > 0 (iffic > 0). 
We remark here that we can compute ; and )2 directly using Equation 63.46 and the center of gravity 
of the exponent relation for this simple system. An interesting question is what happens if b > a. In 
ieee 


this case, the top exponent is 5c — 54 and only the top exponent in Oseledec’s theorem is physically 


“realizable” in the sense that there exists a x9 € R? so that 


a 1}. 24 
P{r = =c—-a}>0. 
{reo(ro) = 5¢— 5a} 


1 


54 a.s. for any xo € R*\{0} in this 


Actually, it follows from Feng and Loparo [14,15] that (xo) = 5c _ 
case. 

Motivated by Bellman [7], Furstenberg and Kesten [16], as a starting point, studied the product of 
random matrices and generalized the classical law of large numbers of an iid. sequence of random 
variables. They showed that under some positivity conditions of the entries of the matrices, the random 
variable n~! log ||X, ...X|| tended to a constant almost surely for a sequence of i.i.d. random matrices 
{X;}°—,. Furstenberg [17,18] went one step further and showed that, if X; € G, for all i, where G is a 
noncompact semisimple connected Lie group with finite center, there exists a finite-dimensional vector 
space of functions ((-), rather than just log || - ||) so that n-!W(X,...X1) tended to a constant hy 
(known as Furstenberg’s constant) almost surely as n + +00. These studies also considered the action 
of the group G on certain manifolds, for example, P4—! C R¢ (the projective sphere in R4, that is, P¢~! 
is obtained by identifying s and —s on the unit sphere S?~! in R“). Furstenberg showed that under a 
certain transitivity or irreducibility condition, there exists a unique invariant probability measure v for 
the Markov chain Z, =X, *...*X1 * Zo, which evolves on P4~!, with common probability measure | 
for X; € G (here, “ *” denotes the action of G on P4~!) and for any xo € R@\{0}, the Lyapunov exponent 


= —— 1 
ho (x0) = lim | — log ||Xn* +++ * X1 * xo|| = i i log ||g * z||W(dg)v(dz). (63.50) 
n>oon 
GxPpt-1 
Furstenberg’s works developed deep and significant results and Equation 63.50 is a prototype for 


computing the Lyapunov exponent in most of the later work that followed. The following simple discrete- 
time example due to Has’minskii best illustrates Furstenberg’s idea. 


Example 63.4: Has’minskii 


Consider a discrete time system, Xp41 =An()Xn, with {An(o)}P° 4, a sequence of i.i.d. random 
matrices. Let 


{én =log||xn|l, and 
On = Ixnll~ xn € S91. 
Then, 


n 


Pn = 10g [IXnll = Pn—1 + 10g llAnPn—1ll = Po + Y. 1Og AK 4-1 I 
k=1 
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Because {An}P°, are independent, {An,@n—1}P°, forms a Markov chain in RIXd x sd-1 if 
{An@n—1}72, is ergodic, the law of large number gives 


_ ee | ip 
Kobo) = i, * on = tim * = log Agi1l| = Etlog |IAgIl} 
i= 
= / / log |Agllu(dA)vidg) as. 


Rdxd yx sd-1 


where it is the common probability measure of {Ai}, and v is the unique invariant probability 


measure of @p with respect to 1, (i.e., v is such that if @o is distributed according to v on sd then 
1 = |IA1 oll '!A1 Gq has the same distribution v on S4—1). 


Has’minskii [20] generalized Furstenberg’s idea to a linear system of Ito stochastic differential equations 
and obtained a necessary and sufficient condition for the almost sure stability of the system: 


dx(t) = Ax(t) dt + du B x(t) d&;(t), t>0 (63.51) 


x(0) = xo 


where &;(f) are independent standard Wiener processes. The process x(t) is then a Markov diffusion 
process with the infinitesimal generator 


1 g au 
Lu= (Ax, uy) + > Xu 998) sag (63.52) 
where u is a real-valued twice continuously differentiable function, uy = Ou/Ox, (-, -) is the standard inner 
product on R4, and 


m 
D(x) = (4(%))axa = > Byxx'Bi (63.53) 
i=1 
By introducing the polar coordinates, p = log ||x|| and @ = ||x||~!x for x 4 0, and applying Ito’s differen- 
tiation rule, 


dp(t) = Q(g(t)) dt + D> g(t) Big(t) déi(t) (63.54) 
i=1 

dep(t) = Ho( g(t) dt +) Hj(¢) d&j(0) (63.55) 
j=l 


where ‘ 
Q(e(t)) = g(t)’Ag(t) + 5 fuel) — (ty X(¢(4)) e(t) 


and Ho(9(t)), Hj(@(t)) are projections of the linear vector fields Ax and B;x on R@\{0} onto S4~!. Then 
from Equation 63.55 we see that (tf) is independent of p(t) and ¢(t) is a Markov diffusion process on 
S4—!. Has’minskii assumed a nondegeneracy condition of the form, 


(H) oa X(x)a> mlilall-llxl], VaeR” and xeR4 


that guaranteed that the angular process ¢(t) is an ergodic process on S4~! with unique invariant proba- 


bility measure v. Then the time average of the right-hand side of Equation 63.54 is equal to the ensemble 
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average, that is, 
_ aera ee i 
Foo) = Tim > [acon de=B(00) =f Qewvde=F as. (6356) 
t>+o t Jo gd-1 


independent of xp € Ré\{0}. 


Theorem 63.7: Has’minskii 


Under condition (H), J < 0 implies that the system Equation 63.51 is almost surely stable. If] > 0, then the 
system Equation 63.51 is almost surely unstable and 


P{ lim ||x(t, xo, )|| = -+oo} = 1. 
t++00 
If J =0, then 


P | lim ||x(t, xo, w)|| = +00} > 0, 
t—+00 


P| lim ||x;(xo, @)|| < +00} > 0. 


t>+00 
To determine the sign of J, the invariant probability measure v must be found. This can be accomplished 
by solving the so-called Fokker—Planck (forward) equation for the density 1 of the measure v. For 


simplicity, we consider the two dimensional case (d = 2). It can be shown from Equation 63.55 that the 
angular process ¢(t) satisfies the Ito equation 


de(t) = B(g(t)) dt + (p(t) dxi(t) (63.57) 


with x;(f) a standard Wiener process and ® and W appropriate functions of ¢(t). p(t) has generator 
d 1 d 
Lo = (9) — + -W?(¢)—. 63.58 
@ (@) a5 TNs (ae ( ) 


Then, the Fokker-Planck equation for \ is 


2 


ld d 
Lie pagel Ow) - Jp PMH) =0 (63.59) 


with normalization and consistency constraints 


2m 


i @de= ty .wO= OR: (63.60) 
0 


Condition (H) guarantees that Equation 63.59 is nonsingular in the sense that ¥(g) 40 for any 
g € S4—, and thus admits a unique solution satisfying Equation 63.60. However, even for simple but 
nontrivial systems, an analytic solution for the invariant density 1 is very difficult to obtain. The problem 
is even more complicated if (H) is not satisfied, and this is more often than not the case in many 
applications. 

In this situation, singularity on S! isa result of Y(@) = 0 for some ¢ € S! and the Markov diffusion pro- 
cess p(t) may not be ergodic on the entire manifold S!. One must then examine each ergodic component 
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of S! to determine a complete description of the invariant probability measure. The law of large numbers 
is then used to compute the exponent for each x9 belonging to the ergodic components. Has’minskii 
[19-22] presented two second-order examples to illustrate the treatment of ergodic components and 
the usefulness of the theorem. In one of the examples it was shown that an unstable linear determinis- 
tic system could be stabilized by introducing white noise into the system. This settled a long standing 
conjecture about the possibility of stabilizing an unstable linear system using noise. Has’minskii’s work 
was fundamental to the later developments of the Lyapunov exponent method for studying stochastic 
stability. 

Following Has’minskii, Kozin and his coworkers studied the case when the nondegeneracy condi- 
tion (H) fails. This work made extensive use of one-dimensional diffusion theory to obtain analytical 
and numerical results for second-order linear white-noise systems. Kozin and Prodromou [28] applied 
Has’minskii’s idea to the random harmonic oscillator system in the two forms, 


du(t) 4 
7 + (1+ oB(t))u(t) = 0, (63.61) 
=“ du(t) du(t) 
ult ult . 
ai +26. = +(1+oB(t))u(t) = 0, (63.62) 


where B(t) represents a Gaussian white-noise process. After transforming the systems into an Ito rep- 
resentation, they observed that condition (H) is not satisfied with singularities U(+4) = 0. Kozin and 
Prodromou then used one-dimensional diffusion theory to study the singularities and sample path behav- 
ior of the angular process ¢(t) near the singularities. The angular process traversed the circle in clockwise 
direction and was neither trapped at any point, nor did it remain at a singularity +> or —} for any 
positive time interval. Thus, the angular process ¢(t) was ergodic on the entire circle S!. From the law of 
large numbers, they obtained 


Ty Te 
E {i Q(g(s)) | + E{ f QG(s)) ds} 
0 


Xw(%o) =J = E{Q(g)} = — aaa is 


independent of xp. 7; and Je are Markov times for ¢(t) to travel the right and left half circles, respectively. 
Using the so-called speed and scale measures, they were able to show the positivity of J for Equation 63.61 
for o” € (0, +00). This proved the almost sure instability of the random harmonic oscillator with white 
noise parametric excitation. For the damped oscillator Equation 63.62, they failed to obtain analytical 
results, and the almost sure stability region was determined numerically in terms of the parameters (&, 0). 

Mitchell [42] studied two second-order Ito differential equations where the condition (H) is satisfied. 
By solving the Fokker—Planck equation, he obtained necessary and sufficient conditions for almost sure 
stability in terms of Bessel functions. Mitchell and Kozin [43] analyzed the linear second-order stochastic 
system in canonical form, 


ee =x2(t), and (63.63) 


xp (t) = —w°x1 (t) — 28x2(t) — fi (t)x1 (t) — fa(t)x2(2). 


Here the f;(t) are stationary Gaussian random processes with wide bandwidth power spectral density. After 
introducing a correction term due to Wong and Zakai, Equation 63.63 was written as an Ito stochastic 
differential equation and Has’minskii’s idea was again applied. Because the angular process ¢(f) is singular, 
one-dimensional diffusion theory was used to give a careful classification of the singularities on S! and the 
behavior of the sample path of (ft) near the singularities. The ergodic components were also examined 
in detail. The difficulty in the analytical determination of the invariant measures corresponding to the 
ergodic components of the angular process was again experienced. An extensive numerical simulation 
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was conducted and the almost sure stability regions were represented graphically for different parameter 
sets of the system. In this work, the authors provided a good explanation of the ergodic properties of 
the angular process ¢(t) and also illustrated various concepts, such as stabilization and destabilization 
of a linear system by noise, by using examples corresponding to different parameter sets. Interestingly, 
they provided an example where the sample solutions of the system behaved just like a deterministic 
system. Examples were also used to show that the regions for second moment stability may be small 
when compared with the regions for a.s. stability, that is, moment stability may be too conservative to be 
practically useful. This is one of the reasons why a.s. stability criteria are important. 

A more detailed study of the case when the nondegeneracy condition is not satisfied was presented by 
Nishioka [45] for the second-order white-noise system given by 


dx(t) = Ax(t) dt + By x(t) dé, (t) + Box(t) dé2(t) (63.64) 


with &;(t), i= 1,2, independent standard Wiener processes. Results similar to Kozin and his coworkers 
were obtained. 

Has’minskii’s results were later refined by Pinsky [49] using the so-called Fredholm alternative and 
the stochastic Lyapunov function f(p, ¢) = p+ h(¢) in the following manner. From the system Equa- 
tion 63.51 with the generator Equation 63.52, Pinsky observed that 


Lf (p, p) = Lo(p) + Lelh(g)] (63.65) 


that is, the action of £ on f (p, ~) separates L into the radial part £, and the angular part Cy. Furthermore, 
L,(p) = v(g) is a function of ¢ only, and Ly is a second-order partial differential operator that generates 
a Markov diffusion process on S4—'. If Has’minskii’s nondegeneracy condition (H) is satisfied, then Ly 
is ergodic and satisfies a Fredholm alternative, that is, the equation 


Loh() = g(9) (63.66) 


has a unique solution, up to an additive constant, provided that 


/ g(o)m(de) =0 (63.67) 


sd-1 
with m being the unique invariant probability measure of the process ¢(t) on S¢~!. Define 
q= / v(g)m(dg). (63.68) 
gd-1 
Choose h(¢g) as a solution of £oh(~) = q — v(¢). From Ito’s formula, Pinsky obtained 


t 


ot) + W(G(t)) = po + Mga) + fLflols) 915) ds-+ Mi = po +h ga) + gt-+ My (63.69) 
0 


where 


t 
Mi = [ H(9(8) 4869) 
0 


for an appropriate function H on S4~!, M; isa zero-mean martingale with respect to the o-algebra gener- 
ated by the process {;, t > 0} and : lim 1M; = 0a.s. Thus, upon dividing both sides of Equation 63.69 
—>+00 
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by ¢ and taking the limit as f + ++00, 
Xo(xo) = li : (t)= (63.70) 
(Xo =m )=q as. é 


Loparo and Blankenship [35], motivated by Pinsky, used an averaging method to study a class of linear 
systems with general jump process coefficients 


£(t) = Ax(t) + > yj(t)Bjx(t) (63.71) 
j=l 


with A’ = —A and y(t) = (y(t)... ym(t))! is a vector-valued jump process with bounded generator Q. 
Using the transformation z; = e 4“tx(t), p(t) = log ||z(£)|| = log ||x(£)||, and o(t) = ||z(¢)||~!z(2), the 
system was transformed into a system of equations involving (p(t), g(f)) that are homogeneous in 
y = (y(t)... ym(t))’ but inhomogeneous in time. Thus, an artificial process t(t) was introduced so 
that [(t), p(t), t(t), y(t)] is a time-homogeneous Markov process. Using the averaging method and the 
Fredholm alternative, a second-order partial differential operator £ was constructed. £ is the generator 
of a diffusion process on R x S¢—!. Applying Pinsky’s idea they obtained 


q= i L(p)m(d0) = / v(9)m(d0) (63.72) 


gd-1 gqd-l 


where m is the unique invariant probability measure on S4~! for the diffusion g; generated by Ly. 
Unfortunately, q is not the top Lyapunov exponent for the system and cannot be used to determine the 
a.s. stability properties of the system. This can be easily verified by considering the random harmonic 
oscillator 

u(t) + k°(1 + by(t))u(t) = 0 (63.73) 


with y(t) € {—1,+1} a telegraph process with mean time between jumps )! > 0. The formula for q is 


k? Xb? 


1= 3242) (63.74) 


independent of x9 € R*\{0}. This is only the first term in an expansion for the Lyapunov exponent (xo); 
see the discussion below. Thus Equations 63.72 and 63.74 do not provide sufficient conditions for almost 
sure stability or instability. 

Following Loparo and Blankenship’s idea, Feng and Loparo [13,36] concentrated on the random 
harmonic oscillator system Equation 63.73. By integration of the Fokker—Planck equation, the positivity 
of dw (xo) = was established for anyk, i € (0, +00) and b € (—1, 1)\{0}. Thus, the system Equation 63.73 
is almost surely unstable for any k, d, and b as constrained above. To compute the Lyapunov exponent \ 
explicitly (it is known that Xo (xo) = X = constant a.s. for any x9 € R*\{0}), the following procedure was 
used. As usual, introducing polar coordinates p = log ||x||, g= \|x|| "1x, where x = (ku, is)’, the system 
becomes 


fee = y(t)go((t)) and (63.75) 


o(t) = h(p(t), y(t) 


for some smooth functions gp and h on S !_ The Markov process (p(t), p(t), y(t)) has generator 


0 0 0 
L=Q+Ag-o>+y Eo no | SL, 4+Ly 


where L; = Q+ Aq: 0/0¢ satisfies a Fredholm alternative. When L acts on the function, F = p+h(g)+ 
fily, 9), where fi(y, ~) is a correction term introduced in the work of Blankenship and Papanicolaou, 


Stability of Stochastic Systems 63-25 


see [13] for details, 


(63.76) 


ie =qo+L2h() and 
qo = TLofi(y, p) = T1v(¢)]. 


In Equation 63.76 the correction term is chosen as the solution of Lif; = —ygo(@), and Lf, = v(¢) is 
a function of ¢ only. Here, 7 is the uniform measure on S!, and h(@) is the solution of Ag- dh/d9 = 
[v() — mv(¢)]. It follows that L2h(~) = ygi(~) and a martingale convergence argument gives 


h= li 
t—+-+00 


t 
1 1 < 
een Ree eet te y (POCO 40 his 
t t>-+oo t 
0 


where 1 is the Lyapunov exponent of the system, 


ie = Lah(o(t)) = y(Ogi(o) and ee 


(t) = h(p(t), y(t). 


Noting the similarity between the systems given by Equations 63.75 and 63.77, the above procedure can 
be repeated for Equation 63.77. Hence, 


(oe) 


K=Ggot = got (qi +%2)= >> ae 
k=0 


The absolute convergence of the above series for any k, i € (0, +00), and b € (—1, 1) was proved by using 
Fourier series methods, and a formula for the general term q, was obtained. To third-order (b°), Feng 
and Loparo obtained 


8(k2 +2)  64(k2 + 2)2 k2 402 0 24 rk? + 2 + 9k? 


cab ie Se k*nD® Ops at i lOO 21 
- " 8x 162(2 + k2)2 


25 32 3 = 
? : 63.78 
(get pe) | D4 sol2) 


In the formulas for the Lyapunov exponent obtained by most researchers, for example, Has’minskii 
for Equation 63.56, computation of an invariant probability measure was always required to determine 
the exponent. This is a very difficult problem that was overcome in the work of Feng and Loparo by 
sequentially applying the Fredholm alternative for the simple, but nontrivial, random harmonic oscillator 
problem. Due to the difficulty involved with the analytical determination of the invariant probability 
measure, several researchers have attacked the problem by using an analytic expansion and perturbation 
technique; some of these efforts have been surveyed by Wihstutz [57]. 

Auslender and Mil’shtein [6] studied a second-order system perturbed by a small white-noise distur- 
bance, as given by 


k 
dx*(t) = Bx*(t) +e) ojx*(t) d&j(t)) (63.79) 
j=l 


where §;(t) are independent, standard Wiener processes and 0 < € < 1 models the small noise. Assuming 
that Has’minskii’s nondegeneracy condition (H) is satisfied, the angular process @*(t) = ||x‘(t)|| 7! x*(#) is 
ergodic on S!, and the invariant density 1 satisfies the Fokker—Planck Equation 63.58, and the exponent 
d(€) is given by Equation 63.56. In this case a small parameter € is involved. Auslender and Mil’shtein 
computed X(€) to second order (e”) and estimated the remainder term to obtain an expansion for X(€) to 
order <* as € | 0*. For different eigenstructures of B, they obtained the following results: 
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0 
1 B= k i , a>b, two distinct real eigenvalues. 


2 k 
MOQ =a— 5 Dor! +e4p(6) + pole) 


r=1 


where o; = (o//)2 x 2, |p(e)| < m < co and |po(e)| < ce~“!/© for some constants c and cy > 0. 


2. B= Be , a,b>0,acomplex conjugate pair of eigenvalues. 
es 12 21\2 1 22\2 4 
MO=at >> [(or —o7')" + (of! +.07*) |+« Rie) 
r=1 


where |R(e€)| < m < +00. 


0 
3. B= E i , one real eigenvalue of geometric multiplicity 2. 


2m 


(eae L(p)n(9) de 
0 


where 
k 


k 
1 
L(g) = 5 DlorA(g), ord()) = D Nor d(), M))?s 


r=1 r=1 
d(¢~) = (cos ¢, sin ¢)’, and 1(@) is the density determined by a Fokker—Planck equation and is 
independent of €. 


1 
4. B= i 1 , one real eigenvalue of geometric multiplicity 1. 


Mosater ren") +0. 
a T(1/6) | 4 P 


r=1 


where I(x) is the gamma function and O(€) denotes a quantity of the same order of € (i.e., 
lime-+9 O(€)/€ = constant). 


In the above work, an intricate computation is required to obtain \(€) as an expansion of powers 
of «. A much easier way to obtain an expansion is the direct use of perturbation analysis of the linear 
operator associated with the forward equation £7. = 0. Pinsky [50] applied this technique to the random 
oscillator problem, 


u(t) + ty + oF (§(t))}u(t) = 0 (63.80) 
where y is a positive constant that determines the natural frequency of the noise-free system, o is a small 


parameter that scales the magnitude of the noise process, and &(f) is a finite-state continuous time Markov 
process with state space M = {1,2,...,N}. F(-) isa function satisfying E{F(&(t))} = 0. After introducing 
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polar coordinates in the form u,/y = pcos ¢ and u = p sin ¢, Pinsky obtained 


F 
p(t) = h(e(t), &(t)) = —/y + ome cos” 9(t) 
(63.81) 
p/o = q(o(t).&(0)) = a sin 29(t) 
where (¢(f), &(£)) is a time-homogeneous Markov process with generator, 
6) 
L= Q cae ry 


and Q is the generator of the process &(t). The Fokker—Planck equation for the density po(@, &), the 
invariant probability density of (p(t), &(t)), is given by 


0=L* po 
where £* is the formal adjoint of £. The exponent is computed by 
Mo) =f alost)palo.8) dds. (63.82) 
SIxM 


Note that 


F 0 
6) cos” O) s Lot+oL}. 


(6) (6) 
L=Q+h—=Q-J/yx—+0 
dg WM Gt ge i 


Assume an approximation of p, in the form 
Po=poto'pit::-+0"pn. 
Then, it follows from 


O= Lipo = (Li +.0L*)(po tolpr +-+++onpn) (63.83) 


that 
Lopit Lipi-1=0, fori=1,2,...,n, (63.84) 


by setting the coefficients of the term o’ in Equation 63.83 equal to zero. Pinsky showed that the expansion 
satisfying Equation 63.84 can be obtained by taking 


po=— and 1 Pn=0 (63.85) 
SIxM 


and proved the convergence of the series expansion by using properties of the finite-state Markov process 


to show that 
+1 


max |Lipyl. 


IPs — (po +o" pi +++-+0"pn)| < co” 
By evaluating p; from Equations 63.84 and 63.85, Pinsky obtained 


2 N 2 
o Ak < Fis We > 3 a 
h(o) = +O(o0°), of 0 63.86 
(0) rs ) ME Ay (0°), of ( ) 
k=2 
where i is the mean sojourn time of &(t) in state k and {j;i= 1,2,...,N} are normalized eigenfunc- 


tions of the linear operator Q. For the case when F() = & is a telegraph process, Pinsky obtained a refined 
expansion of Equation 63.86 consisting of the first two terms in Equation 63.78 when o | 07. 
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Contemporaneous with Pinsky, Arnold, Papanicolaou, and Wihstutz [4] used a similar technique to 
study the random harmonic oscillator in the form, 


—y(t)+oF (s (<)) y(t) = yy(t), (63.87) 


where y, o and p are parameters modeling the natural frequency, the magnitude of the noise, and the 
time scaling of the noise. &(t) is only assumed to be an ergodic Markov process on a smooth connected 
Riemannian manifold M, and F(-) is a function satisfying E{F(&(t))} = 0. A summary of their results is 
given next. 


1. Small noise(o | 0*, y>0, p=1): 
Mo)= PT F2V¥)+0(0%), «Lot 
¥ 


where f (a) is the power spectral density of F(E;). 
2. Large noise (o } +00, y=yotoy1, p=1): 


F 
io = i i i] v(dé) Alain — FO) 00s" 9) +0 ( Bs Gate 


F(&) cos rai 


where Q is the generator of € and v is the unique invariant probability measure of § on M. 
3. Fast noise (p | 0*, o and v fixed): if y > 0, 


om 4 5 ea 
MND O agel wee ee p{0', and 


ify <0, 


Xp) = J+ 5 “Fo O(p*), por. 


4. Slow noise (p t +00, o and y fixed): if y > oe 


1 
=FOarg ey Tee wlte (*) , 


2m 
00) == fag f (de) MEPS 
p 4m y 
0 


ify <omin(F), 


M0) = / v(dt) (oF) — y+ ; / v(de)Q[ (log sin(y + W(E))] 
M 


1 
+0 (=) » ot +0, 
p 


where 1)(&) = tan~! ((oF(€) — y)/./—7) . 


Pardoux and Wihstutz [47] studied the two-dimensional linear white-noise system Equation 63.79 
by using similar perturbation techniques. Instead of using the Fokker—Planck equation, perturbation 
analysis was applied to the backward (adjoint) equation and a Fredholm alternative was used. Pardoux 
and Wihstutz were able to obtain a general scheme for computing the coefficients of the series expansion 
of the exponent for all powers of ¢. Their results are the same as (1), (2), and (3) of Auslender and 
Mil’shtein, if only the first two terms in the expansion are considered. There is another approach for 
studying the problem based on a differential geometric concept and nonlinear control theory, that has 
been suggested by Arnold and his coworkers. For a linear system, after introducing polar coordinates, 
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the angular process ¢(t) is separated from the radial process and is governed by the nonlinear differential 
equation 
O(f) = hE), 9(), ote Ss (63.88) 


where &(t) is the noise process and h is a smooth function of € and ¢. To determine the Lyapunov exponent 
Xeo(xo), the ergodicity of the joint process (E(t), p(t)) needs to be determined. This question is naturally 
related to the concept of reachable sets of the random process ¢(t) and invariant sets of Equation 63.88 
on S4~! for certain “controls” &(t). It is from this perspective that geometric nonlinear control theory is 
applied. 

Arnold et al. [3] considered the system given by 


X(t) = A(E(t))x(t) and 
63.8 
es =x) eR? ( °) 
where A : M — R@*4 is an analytic function with domain M, an analytic connected Riemannian manifold 
that is the state space of a stationary ergodic diffusion process &(t) satisfying the Stratonovich equation, 


d&(t) = Xo(E(t)) dt+ D> X;(E(t)) d&j(0) (63.90) 


jl 


where &;(t) are independent, standard Wiener processes. The following Lie algebraic conditions on the 
vector fields were imposed by Arnold et al.: 


a. dim LA(X},...,X;)()=dimM, V&EeM. 
b. dim LA(h(E,-),& €M)(y)=d-1, VoePt!, 


Here h(é, ys) is the smooth function given in Equation 63.89 for the angular process, and P4~! is the 
projective sphere in R. Condition (a) guarantees that a unique invariant probability density p of € on 
M exists solving the Fokker-Planck equation Q*p = 0, with Q the generator of &(t). Condition (b) is 
equivalent to an accessibility condition for the angular process ~ governed by Equation 63.88 or the fact 
that the system group, 


G= [T12 ,etAG60, t; € R,&; € M, n finite | : 
acts transitively on S¢—!, that is, for any x, y € S4~1, g € Gexists so that g * x = y. Under these conditions, 
they showed that a unique invariant control set C in P4! exists and that this is an ergodic set; all 
trajectories y(t) entering C remain there forever with probability one. Henceforth, a unique invariant 
probability measure 1 of (£, @) on M x P4~! exists with support M x C. The following result was given. 


Theorem 63.8: Arnold, Kilemann, and Oeljeklaus 


For the system defined by Equations 63.89 and 63.90, suppose (a) and (b) above hold, then 


1. 
= f ab oulat.do) 46.9) = 6 Aee (63.91) 
MxC 
is the top exponent in Oseledec’s theorem. 
2. For each 


x9 40, Ko(xo) = das. (63.92) 
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3. For the fundamental matrix P(t) of Equation 63.89, 


. 1 
tim, A log || ®(t)|| =% as. (63.93) 


Equation 63.91 is in the same form as given by Has’minskii. Note that, even though Oseledec’s theorem 
states that only (xo) =} = Amax» the top exponent, is realizable, that is, with probability one can be 
observed from sample solutions, the difficulty is still in determining the invariant density jw. 

Arnold [1,3], motivated by the results for the undamped random oscillator obtained by Molchanov [44] 
studied the relationship between sample and moment stability properties of systems defined by Equa- 
tions 63.89 and 63.90 and obtained some very interesting results. Besides conditions (a) and (b) above, 
another Lie algebraic condition was needed to guarantee that the generator CL of (&, ¢) is elliptic. Define 
the Lyapunov exponent for the pth moment, for p € R, as 


— 1 
8(P»X0) = lim + +00 = log E(\|x(t, x0, w)||P), xo € R@\{0}. (63.94) 


By Jenson’s inequality g(p, xo) is a finite convex function of p with the following properties: 
1. | g(p,x0) |S k| pl, k= max ||A(§)|I. 
&eM 


2. g(p,xo) = AP» Ko(xo) =X as. 
3. g(p,xo)/p is increasing as a function of p with xo € R“\{0} fixed. 
4. g/(0-,x0) < 4 <g’(0*, xo), here, g’ denotes the derivative of g with respect to p. 


Linear operator theory was then used to study the strongly continuous semigroup T;(p) of the generator 
L(p) 2 L+pq(é, ~). Here, £ is the generator of (&, y) and q(E, y) = y'A(E)@. Arnold showed that g(p, xo) 
is actually independent of xo, that is, g(p, xo) = g(p), and g(p) is an analytic function of p with g’(0) = 
and g(0) = 0. g(p) can be characterized by the three figures shown in Figure 63.2 (excluding the trivial 


case, g(p) = 0). 
If g(p) = 0, then \ = 0. If g(p) £0, besides 0, g(p) has at most one zero po # 0, and it follows that 


1. X¥X=Oiff g(p)>0, Vp 0. 
2. > Oiff g(p)<0, forsomep <0. 
3. X<Oiff g(p)<0, forsomep > 0. 


gp) a(p) 
is h=¢ (0) =0 A=” (0) >0 
P 0 P 
gp) 
A=" (0) <0 
0 Pp 


FIGURE 63.2 pth moment Lyapunov exponent. 
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In the case trA(€) = 0, more information can be obtained for the second zero po 4 0 of g(p). Under 
conditions that &(t) is a reversible process along with reachability of ¢(t) on P4~!, Arnold showed that 
po = —d, where d is the system dimension. However, if trA(&) #0, then the system Equation 63.89 is 
equivalent to 

x(t) = d“"trA(&(t))x(t) + Ao(&(t)) x(t) (63.95) 


where trAg(E(t)) = 0. Observing that d—'trA(&(£))I commutes with Ao(&(t)), it follows that 


g(p) = aop + gop) (63.96) 


where a = d~!tr{EA[E(0)]} and g0(p) is the exponent for the pth moment of j(t) = Ao(E(t))y(t). There- 
fore, go(—d) = 0. Applying the results to the damped harmonic oscillator, 


y(t) + 2By(t) + (1+ €(£)) y(t) = 0, (63.97) 


with x = (y,)’ € R*, Arnold obtained 


8(p) = —Bp + go(P) (63.98) 
where go(p) is the moment exponent for the undamped oscillator with go(—2) = 0. Thus, it follows that 


1. For the undamped oscillator Equation 63.97, if B = 0, then \ > 0 and g(p) > 0 for p > 0. This 
implies almost sure sample instability and pth moment instability for p > 0. 

2. The undamped harmonic oscillator can be stabilized by introducing positive damping so that with 
> 0 and g(p) < 0 for p € (0, p1), where p, is a positive real number. 


We remark here that the random harmonic oscillator problem has attracted considerable attention from 
researchers and Equation 63.97 occurs in many science and engineering applications, such as mechanical 
or electrical circuits, solid state theory, wave propagation in random media, and electric power systems. 
It is also of particular interest from a theoretical viewpoint because of the simple structure of the model. 
From the result (1) above, we see that pth moment stability for some p > 0 will imply almost sure sample 
stability. One natural question to ask is the converse question, that is, does almost sure sample stability 
have any implication for the pth moment stability? The result (2) above for the random oscillator gives 
a partial answer to this question. One may ask whether p, in (2) can be arbitrarily large or, equivalently, 
when does the sample stability (. < 0) imply pth moment stability for all p > 0? If 


Ay +X 1 
0<y =e pe < +00, 


then we may choose f > y sufficiently large so that g(p) < 0 for all p > 0. Hence, y is the quantity that 
characterizes the the property that is of interest. 

Another interesting question in the stability study of stochastic systems is when can an unstable linear 
system be stabilized by introducing noise into the system? This question has attracted the attention of 
researchers for a long time. Has’minskii [20] presented an example giving a positive answer to the question. 
Arnold et al. [2] presented a necessary and sufficient condition for stabilization of a linear system x(t) = 
A(t)x(t) by stationary ergodic noise F(t) € IR¢™4 in the sense that the new system x(t) = [A(t) + F(t)]x(t) 
will have a negative top exponent and hence is almost surely stable. This result is presented next. 


Theorem 63.9: Arnold, Crauel, and Wihstutz 


Given the system x(t) = Ax(t) with A € R4*4 a constant matrix, the perturbed system is given by x(t) = 
[A+ F(t)]x(t) where F(t) is a stationary, ergodic, and measurable stochastic process in R¢X4 with finite 
mean. Then 
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1. For any choice of F(t) with E{F(t)} = 0, the top Lyapunov exponent \maxl[A + F(t)] of the perturbed 
system satisfies 
d-"tr(A) < dmax(A + F(t). 


2. For any > 0 fixed, an F(t) exists with E{F(t)} = 0 so that 
d—'tr(A) < dmax[A + F(t)] < d~'tr(A) +e. 


In particular, the linear system x(t) = Ax(t) can be stabilized by a zero-mean stochastic process if, and only 
if, tr(A) <0. 


Note that (2) above implies that the undamped, random harmonic oscillator cannot be stabilized by a 
random linear feedback control of the ergodic type. 

Li and Blankenship [34] generalized Furstenberg’s results on the product of random matrices and 
studied linear stochastic system with Poisson process coefficients of the form, 


dx(t) = Ax(t)dt+ Se Bx(t) dNj(t), (63.99) 
i=1 


where N;(t) are independent Poisson (counting) processes. The solution of Equation 63.99 can be written 
as a product of iid. matrices on R@*@ acting on the initial point x9 € R“\{0} : 


x(t) = exp [A(t = tri) Dixy ...)Dz, exp(At1)x0, 


for t € [twa), tw+1)). Where D; = 1+ B;, for i=1,2,...,m, and N(t) = Ni(t)+---+Nm(t) is also a 
Poisson process with mean interarrival time wh {ts j= 1} are the interarrival times of N(t), tj 2 
T +:-+-+ 1; are the occurrence times of N(t). 

Using the fact that {Xj(@) = D,, exp(A;, aa is an i.i.d. sequence, they generalized Furstenberg’s results 
from semisimple Lie groups to general semigroups (note that D; may be singular) in the following sense. 
Let M = S4-! U {0}, and let \ be the probability distribution of X; induced by 


wl) = P{Dy, e441 ef: Pe B(R**4)} 


where B(R4*“) is the Borel o algebra on R¢*. Let v be an invariant probability measure on M with 
respect to jt, that is, if x9 € M, and is distributed according to v, then x; = || X,x0|| 7! Xyxo has the same 
distribution v. If Qo denotes the collection of so-called extremal invariant probabilities on M with respect 
to w, then Li and Blankenship obtained the following result: 


Theorem 63.10: Li and Blankenship 


For allv € Qo, 
ue oo 
ry 2 oy wf / log ||D; exp(Af)s|je~™! dtv(ds) < +00 
i=l YM 0 


and 


= 1 t, 
Folie) = lim Log Ht 20)ll 


i “Ty as. 
t>-+o0 t Ilxoll 


for all xo € EX, where rr is the mean interarrival time of N;(t),—! is the mean interarrival time of 
Nz, and E2 is an ergodic component corresponding to v € Qo. There are only a finite number of different 


values of ry, say, 11 <2 <+++<1e, €<d. Furthermore, if EQ contains a basis for 4, then the system 
vEQo 
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Equation 63.99 is asymptotically almost surely stable if re < 0, and Equation 63.99 is almost surely unstable 
ifr, > 0. In the case where r; < 0 and re > 0, then the stability of the system depends on the initial state 
xo € R4\ {0}. 


Here, the difficulty is still in determining the extremal invariant probabilities. Li and Blankenship also 
discussed large deviations and the stabilization of the system by linear state feedback. 

Motivated by Oseledec and Furstenberg’s work, Feng and Loparo [14,15] studied the linear system 
given by 
x(t) = A(y(t))x(t),f>0, and 
ie eae (63.100) 


where y(t) € N = {1,2,...,n} is a finite-state continuous time Markov process with infinitesimal gener- 
ator 


Q= (qij)nxn 


and initial probabilities (p1,..., Pn) , and A(i) = A; € R¢**. By using properties of finite-state Markov 
processes and a sojourn time description of the process y(t), they were able to relate the Oseledec spaces 
of the system to invariant subspaces of the constituent linear differential equations x(t) = Ajx(t) and 
obtained a spectrum theorem for the Lyapunov exponents of the stochastic system Equation 63.100. The 
exponents in the spectrum theorem given below are exactly the exponents given in Oseledec’s theorem 
that are physically realizable. It was also shown that the spectrum obtained is actually independent of the 
choice of the initial probabilities (p1,..., pn) of y(t). Thus, stationarity is not required. The theorem is 
stated next. 


Theorem 63.11: Feng and Loparo 
For the system Equation 63.100, suppose p; > 0 and qi > 0 for alli,j ¢ N = {1,2,...,n}. Then there exists 
k real numbers, k < d, constants );,1=1,...,k with 
—0O <p <2 <7 <A < +00, 
an orthonormal basis for R4 
(1) (1) | ,(2) (2) 
fe; aces let sauey, 


fe... ] 


with ij ++--+ ix =d andi; > 1 forj=1,2,...,k so that, if 
Eli = span te)...” 


4 


is an ij-dimensional subspace of R4 and 


Lo = {0} 
LS Op EY f= ls2ss ek 


which is a filtration of R4, that is, 


(lS CeL eel SR? 


where “@®” denotes the direct sum of subspaces, then 


1. Lj={xe R4\ {0} : Xo(x) < hj as. }and Q(x) =rj as. iffx € Lj\Lj-1 forj =1,2,...,k. 
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2. Lj is an Aj-invariant subspace of R4 for alli =1,2,...,nandj =1,2,...,k. 

3. All of the results above are independent of the initial probabilities (p1,...;, Pn) chosen. 

4. If y(t) is stationary, dj is the top exponent in Oseledec’s theorem. 

Stability results can be obtained directly from the theorem, for example, if 4; < 0 and dj+41 > 0, then 


P{ lim ||x(t,x0,@)|| =0}= 1, ifxo € Li, 
t—+-+00 


and 
P{ lim ||x(t,x0,@)|| =+oo}=1, ifxo E R4\L;. 
t—-+00 


63.4 Conclusions 


In this chapter we have introduced the different concepts of stability for stochastic systems and presented 
two techniques for stability analysis. The first extends Lyapunov’s direct method of stability analysis 
for deterministic systems to stochastic systems. The main ingredient is a Lyapunov function and, under 
certain technical conditions, the stability properties of the stochastic system are determined by the “deriva- 
tive” of the Lyapunov function along sample solutions of the stochastic system. As in the deterministic 
theory of Lyapunov stability, a major difficulty in applications is constructing a proper Lyapunov function 
for the system under study. 

Because it is well known that even though moment stability criteria for a stochastic system may be easier 
to determine, moment stability does not necessarily imply sample path stability and that moment stability 
criteria can be too conservative to be practically useful, it is important to determine the sample path 
(almost sure) stability properties of stochastic systems. It is the sample paths, not the moments, that are 
observed in applications. Focusing primarily on linear stochastic systems, we presented the concepts and 
theory of the Lyapunov exponent method for sample path stability of stochastic systems. Computational 
difficulties in computing the top Lyapunov exponent, or its algebraic sign, must still be resolved before 
this method will see more widespread applications in science and engineering. 


Dedication 


The chapter is dedicated to the memory of Dr. Xiangbo Feng, my former student, colleague, and friend, 
who passed away since the first version of this chapter was published. 
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Another important and commonly used class of systems is the class of continuous-time linear systems. 
The models are assumed to evolve in continuous time rather than discrete time because this assumption 
is natural for many models and it is important for the study of discrete-time models when the sampling 
rates are large and for the analysis of numerical round-off errors. The stochastic systems are described by 
linear stochastic differential equations. It is assumed that there are complete observations of the state. 

The general approach to adaptive control that is described here exhibits a splitting or separation of the 
problems of identification of the unknown parameters and adaptive control. Maximum likelihood (or 
equivalently least-squares) estimates are used for the identification of the unknown constant parameters. 
These estimates are given recursively and are shown to be strongly consistent. The adaptive control 
is usually constructed by the so-called certainty equivalence principle, that is, the optimal stationary 
controls are computed by replacing the unknown true parameter values by the current estimates of these 
values. Since the optimal stationary controls can be shown to be continuous functions of the unknown 
parameters, the self-tuning property is verified. It is shown that the family of average costs using the 
control from the certainty equivalence principle converges to the optimal average cost. This verifies the 
self-optimizing property. 

A model for the adaptive control of continuous-time linear stochastic systems with complete 
observations of the state can be described by the following stochastic differential equation 


dX(t) = (A(a)X(t) + BU(t))dt + dW(t) (64.1) 
where X(t) € R”, U(t) € R™. 
P 
A(a) = Ap + 7s aA; (64.2) 


i=1 
Aj € £(R") i=0,...,p, Be L£(R”,R”), (W(t), t € R:) is a standard R”-valued Wiener process and 
Xo =a € R". It is assumed that 


(3.41) A C R? is compact anda € A. 
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(3.A2) (A(a), B) is reachable for each a € A. 
(3.A3) The family (Aj, i= 1,...,p) is linearly independent. 


Let (F;,t € R) be a filtration such that X; is measurable with respect to F; for all te R, and 
(W(t), F;,t € R+) is a Brownian martingale. The ergodic, quadratic control problem for Equation 64.1 
is to minimize the ergodic cost functional 


lim sup “I%o, U, a, t) (64.3) 
t-oo 
where 
t 
(Xo, Usa t) = [ [(QX(s), X(s)) + (PU(S), U(s))] ds (64.4) 


and t € (0, oo], X(0) = Xo, Qe L(R”), and P € £(R”) are self-adjoint and P—! exists, (X(t), t € R4) 
satisfies Equation 64.1 and (U(t),t € R1) is adapted to (F;,t € R;). It is well known [9] that if a is 
known then there is an optimal linear feedback control such that 


U*(t) = KX(t) (64.5) 
where K = —P~!B*V and V is the unique, symmetric, nonnegative definite solution of the algebraic 
Riccati equation 

VA +A*V — VB*P"'BV+Q=0. (64.6) 


For an unknown a the admissible adaptive control policies (U(t), t € R+) are linear feedback controls 
U(t) = K(t)X(t) = K(t, X(u), u<t— A)X(t) (64.7) 


where (K(f), t > 0) is an £(R", R”)-valued process that is uniformly bounded and there is a fixed A > 0 
such that (K(t), tf > 0) is measurable with respect to o(X,,, u < t — A) for each t > A and (K(t), t € [0, A)) 
is a deterministic function. For such an adaptive control, it is elementary to verify that there is a unique 
strong solution of Equation 64.1. The delay A > 0 accounts for some time that is required to compute the 
adaptive control law from the observation of the solution of Equation 64.1. 

Let (U(t), t > 0) be an admissible adaptive control and let (X(t), t > 0) be the associated solution of 
Equation 64.1. Let A(t) = (aj(t)) and A(t) = (aj(t)) be L(R?)-valued processes such that 


t 
ay(t) = [ (Ax@,AXC)ds 
0 
s aj(t) 
aj(t) = ——. 
2 aji(t) 
To verify the strong consistency of a family of least-squares estimates it is assumed that 


(3.4) lim inf |det A(t)| > Oa.s. 
—>0o 


The estimate of the unknown parameter vector at time t, @(f), for t > 0 is the minimizer for the 
quadratic functional of a, L(t, a), given by 


£ t 
L(t,a) = — [ (Ala) + BK(S))X(s),dX(9)) + 5 if |(A(a) + BK(s))X(s)/2ds (64.8) 


where U(s) = K(s)X(s) is an admissible adaptive control. The following result [6] gives the strong con- 
sistency of these least-squares estimators. 
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Theorem 64.1: 


Let (K(t), t > 0) be an admissible adaptive feedback control law. If (3.A1-3.A4) are satisfied and ag € A°, 
the interior of A, then the family of least-squares estimates (Q(t),t > 0), where Q(t) is the minimizer of 
Equation 64.8, is strongly consistent, that is, 


Pay (Jim a(t) = a0) =1 (64.9) 
t>oo 
where Qo is the true parameter vector. 


The family of estimates (&(t),t > 0) can be computed recursively because this process satisfies the 
following stochastic equation: 


daé(t) = A7!(t)(A(t)X(t), dX(t) — A(G(t))X(t) dt — BU(t) dt), (64.10) 


where (A(t)x, y) = (Aix, y))i=1,...,p. 
Now the performance of some admissible adaptive controls is described. 


Proposition 64.1: 


Assume that (3.A1-3.A4) are satisfied and that 


1 
lim —(VX(t),X(t))=0 as. (64.11) 
t>oo t 
: 1 f' 2 
lim sup — |X(t)|"ds<oco as., (64.12) 
too ft 0 


where (X(t), t > 0) is the solution of Equation 64.1 with the admissible adaptive control (U(t),t > 0) and 
@ = 9 € K and V is the solution of the algebraic Riccati equation 64.6 with a = ag. Then 


1 
lim inf —J(Xp,U,a9,T)> trV as., (64.13) 
Too T 


If U is an admissible adaptive control U(t) = K(t)X(t) such that 


lim K(t) = ko a.s., (64.14) 
too 
where ky = —P~!B*V, then 
1 
lim —J(Xo,U,09,T)= trV as., (64.15) 
Too T 


Corollary 64.1: 


Under the assumptions of Proposition 64.1, if Equation 64.14 is satisfied, then Equations 64.11 and 64.12 
are satisfied. 


The previous results can be combined for a complete solution to the stochastic adaptive control problem 
(Equations 64.1 and 64.3) [6]. 
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Theorem 64.2: 


Assume that (3.A1-3.A4) are satisfied. Let (A(t),t >0) be the family of least-squares estimates where 
Q(t) is the minimizer of Equation 64.8. Let (K(t),t>0) be an admissible adaptive control law 
such that 

K(t) = —P7!B* V(&(t — A)) 
where V(q) is the solution of Equation 64.6 for a € A. Then the family of estimates (G(t), t > 0) is strongly 
consistent, 


lim K(t)=ko as., (64.16) 
too 
where ky = —P~!B*V (ao) and 
1 
lim —J(Xp,U,a9,T)=trV as., (64.17) 
Too T 


Now a second formulation for the stochastic adaptive control of an unknown continuous time linear 
system with a quadratic cost is given. In this description a fixed delay A > 0 is introduced for measurability 
and computability of an optimal adaptive control. Let (X(£), t > 0) be a controlled linear diffusion, that 
is, a solution of the stochastic differential equation 


dx(t) = AX(t) dt+ BU(t)dt+CW(t), X(0) = Xo, (64.18) 


where X(t) € R”, U(t) € R”, and (W(t), t > 0) is a standard p-dimensional Wiener process. The prob- 
ability space is (Q,7,P) and (F;,tf > 0) is an increasing family of sub-o-algebras of F such that Fo 
contains all P-null sets, (W(t), F;, t > 0) isa continuous martingale and X(t) € F; for all t > 0. The linear 
transformations A, B, and C are assumed to be unknown. Since the adaptive control does not depend on 
C it suffices to estimate the pair (A, B). For notational simplicity let 6! = [A,B]. 

For the adaptive control problem, it is required to minimize the ergodic cost functional 


t 

lim sup J(t, U) =lim sup : (X7(s)Q1X(s) + U" (s)Q.U(s)) ds (64.19) 
t>0o tooo Ft Jo 

where Q,; > 0 and Q> > Oand U is an admissible control. 

Since both A and B are unknown, it is necessary to ensure sufficient excitation in the control to obtain 
consistency of a family of estimates. This is accomplished by a diminishing excitation control (dither) that 
is asymptotically negligible for the ergodic cost functional Equation 64.19. Let (v,,n € N) be a sequence 
of R™-valued independent, identically distributed random variables that is independent of the Wiener 
process (W(t), t > 0). It is assumed that E[v,] = 0, E [Yn ve] =I for all n € N, and there is ao > 0 such 


n 


that ||v,||? <0 as. for alln EN. Lete € (0, 5) and fix it. Define the R”-valued process (V(t), t > 0) as 


aml 
Dl> 


Vv 
V(t) =) a lnans nay) (64.20) 
n=0 


where A > 0 is fixed. A family of least-squares estimates (0(t), t > 0) is used to estimate the unknown 
9 = [A, B]’. The estimate 4(t) is given by 


6(t) = P(t) i (s) dX" (s) + F(t)P-!(0)0(0), (64.21) 
0 


-1 
rt) = (f «or ds-+al) , (64.22) 
0 


g(s) =[X"(s) UTS), (64.23) 
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where 0(0) and a > O are arbitrary. The diminishingly excited control is 
U(t) = U4(t) + V(t) (64.24) 
where U% is a “desired” control. 


For A stable, (A, C) controllable and some measurability and asymptotic boundedness of the control, 
the family of estimates (0(t), t > 0) is strongly consistent [2]. 


Theorem 64.3: 


Let e€ (0, ) be given in Equation 64.20. For Equation 64.18, if A is stable, (A, C) is controllable and the 
control (U(t), t > 0) is given by Equation 64.24, where U4(t) € Fir_ayvo for t > 0 and A > 0 is fixed and 


t 
/ |U4(s)||2ds = O(t!*5) as. (64.25) 
0 


as t > oo for some 8 € [0, 1 — 28) then, 


16 ani? = 0 ( 8") ae (64.26) 


as t — oo for eachae (48, 1— e), where 9=[A BI]! and 0(t) satisfies Equation 64.21. 


Now a self-optimizing adaptive control is constructed for the unknown linear stochastic system (Equa- 
tion 64.18) with the quadratic ergodic cost functional (Equation 64.19). The adaptive control switches 
between a certainty equivalence control and the zero control. The family of admissible controls U/(A) is 
defined as follows: 


U(A) = {u :U(t) = U4(t) + Ul (t), UN(t) € Fu_ayvo and 
Ul(t) € o(V(s),(t- A) V0 <s <2) forall t>0 


t 
|X(£)||? = o(f) a.s. and / (]U(s) ||? + .X(s)|I*) ds 
0 


= O(t) as. as t > oo]. (64.27) 


Define the R’-valued process (U°(t), t > A) using the equation 
U°(t) = —Q;'BT(t — A)P(t— A) 


t 
x (Mx A)+ / elf -AE-A) Be — AYU 4(s) as), (64.28) 
t-A 


where A(t) and B(t) are the least-squares estimates of A and B given by Equation 64.21 and P(f) is the 
minimal solution of the algebraic Riccati equation 


A’ (t)P(t) + P(t)A(t) — P(t)B(t)Q; 1B" (t)P(t) + Qi = 0, (64.29) 


if A(t) is stable and otherwise P(t) = 0. 
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To define the switching in the adaptive control, the following two sequences of stopping times (on, = 
1,2,...) and (ty,n = 1,2,...) are given as follows: 


09 = 0, 


t 
On = sup f >t: i, lI U(r) ||? dr < st, A(s — A) is stable for all s € [ty, of (64.30) 
0 


‘ 1 
Tn = inf f >t-1t1: / |U@IPdr < 5t'** A(t — A) is stable and [x(t — A)? < ae), 
0 


(64.31) 
The adaptive control (U* (tf), t > 0) is given by 
U*(t) = U4(t) + Vit), (64.32) 
where 
u%(t) = ae a 2 “a ees i ce x i (ee) 


The adaptive control U* is self-optimizing [2]. 


Theorem 64.4: 


If A is stable and (A, C) is controllable, then the adaptive control (U*(t), t > 0) given by Equation 64.22 
belongs to U(A) and is self-optimizing for Equations 64.18 and 64.19, that is, 


inf lim sup J(t, U) = lim J(t, U*) = tr(C’ PC) + tr (B’ PR(A)PBQ;') as. (64.34) 
UEU(A)  t-s00 too 


where P is the minimal solution of the algebraic Riccati equation 64.29 using A and B and 
oe Tr 
R(A) = / ec) 2 dt: 
0 


A third adaptive linear-quadratic Gaussian (LQG) control problem is considered now. In this case only 
a delay A > 0 is introduced for the measurability of the adaptive control. This problem is solved in [4]. 
In this case, the unknowns are A and B and there are only the usual assumptions of controllability and 
observability that are made for the LQG control problem when the system is known. Let (X(t), t > 0) be 
the process that satisfies the stochastic differential equation 


dX(t) = AX(t) dt + BU(t) dt + DdW(t), X(0) = Xo, (64.35) 


where X(t) € R", U(t) ER”, and (W(t), F(t),t => 0) is an R?-valued standard Wiener process, and 
(U(t), F(t), t = 0) is a control from a family that is specified subsequently. The random variables are 
defined on a fixed complete probability space (Q, F, P) and the filtration (F(t), t > 0) is defined on this 
space and specified subsequently. It is assumed that the matrices A and B are unknown. 
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The objective is to design an admissible control process (U(t), F(t), t => 0) so that the following ergodic 
cost functional for the system (Equation 64.35) is minimized 


1 rr 
J(O) = limsupr soz i Ix? (QiX(t) + UT (NQUW)I dt, (64.36) 
0 
where Q> > 0 and Q; > 0 are symmetric and a control (U(t), t > 0) for the system Equation 64.35 is said 


to be admissible if it is adapted to (F(t), t > 0) and 


Jo \U(s) Pde 


64.37 
Jo \X(s)l2ds one 


limsupt—oo 


The following standard assumptions are made. 


Al: (A,B) is controllable. 
A2:; (A, Qi! y is observable. 


It is well known that under the assumptions Al and A2, the optimal control for the known system in 
the family of admissible controls is a linear feedback expressed as 


U°(t) = —Q; |B RX(t), (64.38) 
where R is the unique positive, symmetric solution of the following algebraic Riccati equation 
ATR+RA—ABQ;'B'R+Q: =0. (64.39) 
The corresponding minimal cost is 
J(U°) = infyJ(U)=tr(D™RD) as. (64.40) 
To describe the estimation problem in a standard form, let 


eT =[A BI, (64.41) 
b(t)? =[X7(t) UT], (64.42) 


so that Equation 64.35 can be rewritten as a linear regression 
dX(t) = 6" $(t)dt + DdW(t). (64.43) 
Now the faimily of continuous-time, weighted least-squares (WLS) estimates [1], (O(f), t > 0) is given by 


d0(t) = a(t)P(t)(t)AX? (t) — (£)A(t) de], (64.44) 
dP(t) = —a(t)P(t)(t)" (t)P(t) dt, (64.45) 


where 6(0) = [Ao, Bo]! and P(0) > 0 are arbitrary deterministic values such that (Ag, Bo) is controllable 
and (Ao, Qi!) is observable 


t 
a(t) r(t) = ||P~'(0)|| + i \b(s)|* ds, 


1 
~ f(r(t))’ 
and f € F with 


< co for some c > 0} 


F =(flf:Ry > Ry slowly: i me 
= : > s increasing, 

+ + Slowly 8: xf (x) 
where a function f is called slowly increasing if it is increasing and satisfies f > 1 and f (x?) = O( f (x)) as 
x > OO. 
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Remark 


A necessary condition for a function f € F is that f(x) = o(logx)) Some typical functions that are used in 
WLS algorithms are log!**x and (logx)(loglogx)'**. In fact the family of weights (a(t), t > 0) satisfies 


a |(t) =f (r(t)) + O(log*r(t)) 


for some k > 0 as t > oo. 
It can be shown that the convergence rate of the WLS algorithm can be characterized by P(t), that is, 


\|0(t) — Ol]? = OCI P(E)II) 


The explicit solution of P(t) is 


t 
P(t) = [P71(0) + i a(s)o(s)o(s)" ds], 
0 


which clearly shows that P(t) is positive and nonincreasing so (P(t), t > 0) converges a.s. as t + oo. It is 
also worth noting that the standard least-squares algorithm corresponds to the choice f(x) = 1 which is 
excluded for the WLS algorithm. 


Definition 64.1:. 


A family of linear system models (A(t), B(t), t > 0) is said to be uniformly controllable if there is a c > 0 
such that 


t]-1 
y Al (t)B(t)B"(t)A'" (t) > cl (64.46) 
i=0 


for all t € [0, oo). 


A family of models (A(t), C(t), t > 0) is said to be uniformly observable if (A? (t), C!(t), t = 0) is 
uniformly controllable. 

Let (nz, k € N) be a sequence of independent, identically distributed M(n + m, n)-valued random vari- 
ables that is independent, identically distributed M(n + m, n)-valued random varibles that is independent 
of (W(t), t > 0) so that for each k € N the random variable 1, is uniformly distributed on the unit ball for 
a norm of the matrices. The maximization procedure is recursively defined as 


Bo = 0 (64.47) 
Be=nk iff(ks ng) = + yf(k, Be-1)) (64.48) 


= Px_) otherwise 


Finally, the family of continuous time estimates (A(t), t = 0) to be used for the adaptive control problem 
is simply a piecewise constant function induced by Equation 64.44 


O(t)=6,, te (k,k+11, (64.49) 
6, = O(k) — P!/?(k)Bx, (64.50) 


where k € N. 
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The above estimates are expressed as 


67(t) =[AW) BU)]. (64.51) 
It can be verified that the family (A(t), B(f), Qi , t > 0) is uniformly controllable and observable. Thus, 
the following stochastic algebraic Riccati equation: 


A’ (t)B(t) + R(t)A(t) — R(t)B(t)Q5 'B’ R(t) + Qi = 0 (64.52) 


has a unique, adapted, symmetric, positive solution R(t) for each t € [0, 00) a.s. 
Using R(t), define a lagged certainty equivalence LQG control by 


U(t) = —Q;'B"(t)R(t)X(t) (64.53) 


The following theorem states that for the above lagged certainty equivalence control, the solution of 
Equation 64.35 is stable in the averaging sense. 


Theorem 64.5: 
The process (X(t), t > 0) that is the solution of Equation 64.35 is stable in the sense that 


1 T 
limsupt—oo 7 / |X(s)|?ds < 00 a.s., (64.54) 
0 


To obtain the optimality of the quadratic cost functional, it is necessary to obtain the strong consistency 
for the family of estimates (0(t), t > 0). For this, a diminishing excitation is added to the adaptive control, 
Equation 64.53, that is 


U*(t) = LeX(t) + vel V(t) — Vil (64.55) 
or 
dU* (t) = Ly dX (t) + yxd V(t) (64.56) 
for t € (k,k+ 1] andk € N, where U*(0) € R” is an arbitrary deterministic vector, 
log(k) 
Vk 


for k> 1. The process (V(t), f > 0) is an R”-valued standard Wiener process that is independent of 
(W(t),t > 0) and (ng, k € N). Without loss of generality, the sub-o-algebra F(t) is defined as the P- 
completion of o(Xo, W(s), nj, V(s),s < tj < t). 

The following theorem states that the family of regularized WLS estimates is strongly consistent using 
the lagged certainty equivalence control with diminishing excitation [4]. 


Ly =—Q,'B'(K)R(k) and yy (64.57) 


Theorem 64.6: 
Let (A(t), t > 0) be the family of estimates given by Equation 64.49 using the control (Equation 64.55) in 
Equation 64.35. If Al and A2 are satisfied, then 

lim:+00(t)=0 as. (64.58) 


where 0 is the true system parameters. 


64-10 Control System Advanced Methods 


The following theorem states the self-optimality of the diminishingly excited lagged certainty equiva- 
lence control [4]. 


Theorem 64.7: 


Let Al and A2 be satisfied for the stochastic system (Equation 64.35) with the cost functional Equation 64.36 
where A and B are unknown. Then the adaptive control defined by Equation 64.55 is admissible and optimal, 
that is, 


T 
limsupr-+oo i; [x (t)Q) X(t) + U*" (t)Q,U*(t)]dt = tr(D'RD) —ass., (64.59) 
0 


64.1 Adaptive Control for Continuous-Time Scalar Linear 
Systems with Fractional Brownian Motions 


Brownian motion is often used to model some continuous perturbations of many physical systems because 
this process is Gauss—Markov and has independent increments. However, empirical measurements of 
many physical phenomena suggest that Brownian motion is inappropriate to use in mathematical models 
of these phenomena. A family of processes that has empirical evidence of wide physical applicability 
is fractional Brownian motion. Fractional Brownian motion is a family of Gaussian processes that was 
defined by Kolmogorov [13] in his study of turbulence [14,15]. While this family of processes includes 
Brownian motion, it also includes other processes that describe behavior that is bursty or has a long-range 
dependence. The first empirical evidence of the usefulness of these latter processes was made by Hurst [11] 
in his statistical analysis of rainfall along the Nile River. Mandelbrot used fractional Brownian motions 
to describe economic data and noted that Hurst’s statistical analysis was identifying the appropriate 
fractional Brownian motion (FBM). Mandelbrot and van Ness [16] provided some of the initial theory for 
FBMs. Empirical justifications for modeling with FBMs have occurred in a wide variety of phenomena, 
such as, economic data , flicker noise in electronic devices, turbulence, internet traffic, biology, and 
medicine. 

A real-valued process (B(t), t > 0) on a complete probability space (Q, F, P) is called a (real-valued) 
standard fractional Brownian motion with the Hurst parameter H € (0, 1) if it is a Gaussian process with 
continuous sample paths that satisfies 


E[B(t)] =0 


64.60 
B [B(S)B(O)] = 5 (8 +924 — es) (64.60) 


for alls,t e Ry. 

Let H € (1/2, 1) be fixed and B be a fractional Brownian motion with Hurst parameter H. For the 
applications given here, only a few results from a stochastic calculus for fractional Brownian motion are 
necessary. Let f: [0, T] — R be a Borel measurable function. If f satisfies 


2 = H-1/2 2 
lig, = 0 (mp0 |V? (naps) ds <oo, (64.61) 


then f € LZ, and ie f dB is a zero-mean Gaussian random variable with second moment [10] 


=[(f') | =|f,. (64.62) 
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where uq(s) = s“ fora >O ands > 0, ae * is a fractional integral [17] defined almost everywhere and 
given by 
H-1 Ae) f@ 
(Bean a = Fal ak PTE dt (64.63) 


for x € [0, T), f € L1({0, T]), and I'(-) is the Gamma function and 


HT(H + 1/2)0(3/2 —H) 
T(2—2H) ; 


p(H) = 
Consider the optimal control problem where the scalar state X satisfies 
dX(t) = apX(t) dt+ bU(t) dt+dB(t), X(t) =Xo, (64.64) 


and the ergodic (or average cost per unit time) cost function J is 
1 T 
J(U) = lim sup — i (qX?(t) + rU*(t)) dt, (64.65) 
T= 00 T 0 


where q > 0 and r > 0 are constants. The family U/ of admissible controls is all (F;) adapted processes 
such that Equation 64.64 has one and only one solution. 

To introduce some notation, recall the well-known solution with H = 1/2, that is, (B(t),t > 0) is a 
standard Brownian motion. An optimal control is U* given by 


b 
U*(f)=—- ; poX*(t) (64.66) 


where (X*(t),t > 0) is the solution of Equation 64.64 with the control U*, po is the unique positive 
solution of the scalar algebraic Riccati equation 


b2 
—p’? —2ap—q=0, (64.67) 
r 
hence 
; 
Po = plo +8], (64.68) 
b2 
80 =,/a,+—q. (64.69) 
r 
Furthermore, 
J(U*)=po as, (64.70) 


The following result is given in [12] and solves the analogous control problem for H € (1/2, 1). 


Theorem 64.8: 
Let (U*(t), t > 0) be the control given by 


b 
U*(t)= —= PoLX™(t) + v*(o)], (64.71) 
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t t 
ro= | Bave(sas+ f [Res =] (dX*(s) — aX*(s) — bU*(s)) ds 
on e (64.72) 
= i e 0s!) qB(s | t), 
t 


where (X*(t), t > 0) is the solution of Equation 64.64 with the admissible control (U*(t), t > 0), po and 89 
are given in Equations 64.68 and 64.69 respectively, and 


= d rt 

k(t,s) = ee / (r— sl P-Hv(r, r) dr, 
“ Ss 

y(t,s) = toe [ eK (t, s) dt, 


0 


T 
Ku(ts) = HH —1) [ pH-l/2(p — syH—-3/2 dy, 


B(s | t) = E[B(s) | Fi] 


t 
= B+ fant ayy) AB 
0 
t 
=H) + | ton 4-1 ales) dW, (64.73) 
0 


where cy is a constant that only depends on H, ua(s) = s* for s > 0, I#—'/? is a fractional integral (Equa- 
tion 64.63), D-H js the fractional derivative and (W(t), t > 0) is a standard Brownian motion (Wiener 
process) associated with (B(t),t > 0) (e.g., [3]). 

Then the control U* is optimal in U and the optimal cost is 


J(U*) = a.s., (64.74) 


where 


= | eae 


64.75 
maa (64.75) 


89 — Qo 


If ap is unknown, then it is important to find a family of strongly consistent estimators of the unknown 
parameter a in Equation 64.64. A method is used in [7] that is called pseudo-least squares because it 
uses the least-squares estimate for a9 assuming H = 1/2, that is, B is a standard Brownian motion in 
Equation 64.64. It is shown in [7] that the family of estimators (a(t), t > 0) is strongly consistent for 
H € (1/2, 1) where 


t yo 
&(t) = a9 4 J oX (s) cat) (64.76) 
Jy (Xs)? ds 
where 
dX°(t) = ap X°(t) dt + dB(t), X°(0) = Xo. (64.77) 


This family of estimators can be obtained from Equation 64.64 by removing the control term by subtrac- 
tion. The family of estimators & is modified here using the fact that ao € [a1, a2] as 


cu(t) = O()1 fay ,ay] (Q(t) + 1 1 (00,01) (&(E)) + 42 1 (ay,00) (Q(t) (64.78) 


for t > 0. &(0) is chosen arbitrarily in [a), a2]. 
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An adaptive control (U“(t), t > 0), is obtained from the certainty equivalence principle, that is, at time 
t, the estimate a(t) is assumed to be the correct value for the parameter. Thus the stochastic equation for 
the system (Equation 64.64) with the control U% is 


dX’ (t) = (ag — a(t) — 8(t)) X(t) dt — “et V(t) dt + dB(t) 
= (—ay — a(t) — 8(f))X” dt — (a(t) + 8(t)) V(t) dt + dB(t), (64.79) 
X*(0) = Xo, 
and 
b2 
d(t) = 07 (6) 14 (64.80) 
w= 0) [X(t + VO), (64.81) 
p(t) = 5 lat) +80), (64.82) 
t t 
VA(t)= / §(s)V(s) as+ [ [k(t, s) — 1][dX”(s) — a(s)X”(s) ds — bU’(s) ds] 
~ a (64.83) 
=f §(s)V\(s) as+ [ [k(t, s) — 1][dB(s) + (ap — a(t))X”(s) ds], 
0 0 
8(t) = 8(t) + a(t) — a0, (64.84) 


and k denotes the use of 5 instead of So in k. Note that 8(t) > —a(t)+c for some c > 0 and all t > 0 so 
that 
ag — a(t) — 8(t) < -c. 


The solution of the stochastic equation 64.79 is 
be t ‘A 
Hae fo x4 i ew Js 51 (a(s) + 8(s)] V(s) ds + dB(s)]. (64.85) 
0 


The following result [8] provides the solution of an adaptive control problem in the average sense instead 
of the pointwise sense. 


Theorem 64.9: 


Let the scalar-valued control system satisfy the Equation 64.64. Let (a(t), t > 0) be the family of estimators 
of ao given by Equation 64.78, let (U’(t), t > 0) be the associated adaptive control in Equation 64.81, and 
let (X(t), t = 0) be the solution of Equation 64.64 with the control U’.. Then 


t 
lim +B 1 |U*(s) - U%\(s)|” ds =0 (64.86) 
t>o t 0 
and ; 
1 
lim —E i} |x*(s) —X(s)|? ds =0, (64.87) 
t>oo ft 0 
hence : 
in oe / (q(X(s))? + r(U%(s))’) ds =, (64.88) 
t>oo t 0 


where i is given in Equation 64.75. 
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65.1 Motivations and Preliminaries 


In this chapter, we study probabilistic and randomized methods for analysis and design of uncertain 


systems. This area is fairly recent, see [10,11], even though its roots lie in the robustness techniques for 


handling complex control systems developed in the 1980s. In contrast to these previous deterministic 
techniques, the main feature of these methods is the use of probabilistic concepts. One of the goals of 
this methodology is to provide a rapprochement between the classical stochastic and robust paradigms, 


combining worst-case bounds with probabilistic information, thus potentially reducing the conservatism 


inherent in the worst-case design. In this way, the control engineer gains additional insight that may help 
bridging the gap between theory and applications. 
We consider an uncertain system affected by parametric uncertainty 


q=(41 


oa qe! 
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which is bounded in an hyperrectangle of radius p € [0, 1] centered at the nominal value q 
Q,= {ger : qi € [1 — p) Gis (1+ p)ghi=...¢}. 


Henceforth, the objective is to design controller parameters 6 in the presence of uncertain constraints of 
the form f(8, q) < 0; in Section 65.2 we provide a specific example of function f. 

The algorithms derived in this probabilistic context are based on uncertainty randomization and are 
usually called randomized algorithms. That is, assuming that q is a random vector with given probability 
measure Pr, a controller is constructed utilizing a number of random samples q“ of q. These algorithms 
provide controller parameters 9 which probabilistically satisfy the system constraint f (9, q) < 0. In other 
words, a certain probability of violation V (0, q) is associated to the controller 9, but this probability may 
be suitably bounded by given (probabilistic) accuracy € € (0,1) and confidence 8 € (0,1). The results 
presented in the literature for solving probabilistic design of uncertain systems can be divided into two 
main categories consisting of sequential and nonsequential methods, which are now described more 
precisely. 

Sequential methods, see Section 65.3, generally build upon a convexity assumption regarding how 
the parameters 9 enter into the function f(0,q). The resulting algorithms are based on a probabilistic 
oracle and subsequent update rule. Various update rules have been derived, including gradient [3,9,27], 
ellipsoid [7], and cutting plane [4]. In this context, finite-time probabilistic properties have been obtained 
providing bounds on the maximum number of required iterations to guarantee a termination criterion 
with given probabilistic accuracy and confidence. Design of robust Linear Quadratic regulators, synthesis 
of linear parameter-varying (LPV) systems, and solution of uncertain linear matrix inequalities (LMIs) 
are successful examples of the efficacy of these methods. 

Nonsequential methods, see Sections 65.4 and 65.5, are mainly based on statistical learning theory; 
see [13] for further details. From the control design point of view, the objective is to derive sample size 
bounds which guarantee uniform convergence properties for feasibility or optimization problems. The 
methods developed are very general and are not based on any convexity assumption. We present an 
algorithm which requires a one-shot (local) solution of an optimization problem subject to randomized 
constraints, whose number is dictated by the sample complexity previously obtained. 

In the particular case of convex optimization with nonsequential methods, see Section 65.4, a successful 
paradigm has been introduced in [1]. In this approach, the original robust control problem is reformulated 
in terms ofa single convex optimization problem with sampled constraints which are randomly generated. 
The main result of this line of research is to determine the sample complexity without resorting to statistical 
learning methods. 

Once a probabilistic controller has been obtained either with sequential or nonsequential methods, 
the control engineer should perform an a posteriori analysis, both deterministic and probabilistic; see 
Section 65.6. In particular, regarding the former analysis, standard robustness methods can be used. 
Regarding the latter, in a classical Monte Carlo setting, we can establish the sample complexity for analysis 
so that the probability of violation is estimated with given accuracy and confidence. The probabilistic 
margin obtained with this approach is larger than that computed with deterministic methods at the 
expense of a “small” probability of violation. 

To better illustrate these concepts, we resort to a motivating example, which will be revisited in various 
forms in this chapter. 


Example 65.1: Design of a Lateral Motion Controller for an Aircraft 


We consider a multivariable example given in [11] (see also [18] for a slightly different model and set 
of data) which studies the design of a controller for the lateral motion of an aircraft. The state-space 
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equation consists of four states and two inputs and is given by 


0 1 0 0 | 0 0 | 
; 0 Lp Le Ly 0 Ls, 
Ng(g/V) Np Ng + Ng Ye Nr — Nj | Ns, +NBY5, Ns, | 


where xj is the bank angle, x2 its derivative, x3 is the sideslip angle, x4 the yaw rate, u; the rudder 
deflection, and u2 the aileron deflection. 

We consider the case when the 13 aircraft parameters entering into the state and input matrices 
of Equation 65.1 are uncertain. Hence, we consider the system 


X(t) = A(q)x(t) + B(q)u(t), 


where we introduced the uncertainty vector q = [q1 --- ql! with @ = 13. In particular, we consider 
the case when each parameter qj; is perturbed by a relative uncertainty equal to 10% around its 
nominal value gj, as reported in Table 65.1. 

Formally, the vector g is assumed to range in the hyperrectangle centered at the nominal value q, 
that is, 


Q>= {aeRé : gi €[0.9Gj, 1.1 Gj] = Weta hs (65.2) 


We are interested in designing a state feedback controller u = Kx that robustly stabilizes the 
system guaranteeing a desired decay rate « > 0, which is equivalent to having all closed-loop eigen- 
values with real part smaller than —a. A sufficient condition [22] for the existence of such a controller 
requires finding a symmetric positive definite matrix P € R44, and a matrix W € R* such that the 
following quadratic performance criterion is satisfied for all values of g € Qp, 


Pop(W, P,q) = A(q)P + PA' (q) + B(q)W' + WB! (q) + 2aP <0, (65.3) 


where the symbols > and < (> and =) denote positive and negative definite (semidefinite) matrices. 
Further, if we find common P > 0 and W that simultaneously satisfy Equation 65 3 for all gq € Qp, then 
a control gain K robustly guaranteeing the desired decay rate can be recovered as K = W! P—". This 
approach has strong connections with classical optimal control, and in particular guaranteed-cost 
regulator design; see [11,18] for further details. 

It should be noted that in many practical situations the requirement that Equation 65.3 is satisfied 
for all possible values of g may be too strict and may result in an overly conservative design. Suppose 
further that additional information on q is available, namely q has probabilistic nature, that is, it is 
a random vector distributed according to a probability measure Pr. In this case, it is reasonable to 
look for a design that guarantees that the closed-loop system behaves as desired with a given “high" 
probability. More precisely, fixing a priori a (small) probability level « € (0, 1), we look for P > 0, W that 
satisfies the probability constraint 


Pr{q€Qp: Pap(P,W,q) x0} > 1-«. 
Note also that, if we introduce the function 


f(P, W, q) = max (PaplP, W, q)) , 


TABLE 65.1 Uncertain Parameters and Nominal Values 
q- Ip Le Lr g/V Yp_ Na Np Np N; Ls Y5 Ng Ng 


q —2.93 -—4.75 0.78 0.086 -—0.11 O01 —0.042 2.601 -—0.29 —3.91 0.035 —2.5335 0.31 


a rT ir a 
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where Xmax(-) denotes the largest eigenvalue of its argument, then we may rewrite a probabilistic 
problem as that of finding P > 0, W such that 


Pr{q€Qp:f(P,W,q) <0} >1-«. (65.4) 


Computing a solution that satisfies Equation 65.4 amounts to solving a so-called chance-constrained 
semidefinite feasibility problem: a non-convex and very hard problem in general. 


65.2 Probabilistic Design 


The design techniques developed in the probabilistic framework are based on the interplay of random 
sampling in the uncertainty space and deterministic optimization in the design parameter space. Formally, 
let 9 © R™ denote the vector of design variables. For instance, returning to the previous motivating 
example, we choose 6 to represent the free parameters of the symmetric matrix P and the full matrix W, 
so that mg = 18. Then, define a performance function that takes into account all the design and performance 
constraints related to the system with design parameters 9. These are rewritten in the form of the following 
(uncertain) inequality 


f(9,q) =f(P; W,q) <0, (65.5) 
where f(8,q) : Qp x R” — R is a scalar-valued function. The design vector 8 such that the inequality 


(Equation 65.5) is satisfied “for most” (in a probabilistic sense) of the outcomes of q is called a probabilistic 
robust design. More specifically, define the probability of violation of the design 0 as 


V(0) =Pr{qeQ) : f(®,q) >}, 
then, an €-probabilistic robust design is a design guaranteeing 
V(0) <e. 
Equivalently, we define the probability of performance, or shortly reliability, of the design 0 as 
R(0) =1—- V(0) = Pr{qeQ, : f(9,q) < o}. 
In other words, we say that a design is reliable if it guarantees a probability of performance of at least 
(1 —€). Most of the results presented in the literature for solving the probabilistic design problem have 


been derived under the assumption that the function f (0, q) is convex in 0 for all q € Q. This assumption, 
which will be used in Sections 65.3 and 65.4, is now formally stated. 


Convexity Assumption 65.1: 


The function f (9, q) is convex in 0 for any fixed value of q € Qo. 


A standard example of a convex function f arises when considering performance requirements 
expressed by LMI conditions, as in the case of the motivating example. In particular, consider robust 
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feasibility problem of the form 
Find@ such that F(0,q)<x0 VqeEQ,), 


where the constraint F(9, q) < 0 is an LMI in 0 for fixed q, that is, 


ng 
F(0, q) = Fo(q) + >. 6:Fi(Q), (65.6) 
i=l 
and F;(q),i =0,...,n9, are symmetric real matrices of appropriate dimensions which depend in a generic 


and possibly nonlinear way on the uncertainty q € Q,. Then, to rewrite this problem in the scalar-function 
framework, we simply set 


f(9;.q) = kmax(F(6, q)). 


Finally, we remark that considering scalar-valued constraint functions is without loss of generality, 
since multiple constraints 


fi, q) <0,.. -fny (9, 4) <0 


can be immediately reduced to a single scalar-valued constraint by setting 
f(9,q) =. max fj(9, q). 
i=l,....mg 


The randomized algorithms discussed in this chapter provide a numerically viable way to compute 
approximate probabilistic solutions for the problems previously described. 


65.3 Sequential Methods for Design 


In this section, we present randomized sequential methods for finding a probabilistic feasible solution 6 
to the uncertain inequality 


f(@,q) <0 (65.7) 


introduced in the previous section, under the Convexity Assumption. To this end, we first state the 
following formal definition. 


Definition 65.1: r-Feasibility 


For given r > 0, we say that the inequality (Equation 65.7) is r-feasible if the solution set 
S={0ER™ : f(0,q) <0,Vq € Qo} 
contains a full-dimensional ball of radius r. 


The algorithms presented in the literature for finding a probabilistic feasible solution (i.e., a solution 
that lies in S with high probability) are based on two fundamental ingredients: (1) an Oracle, which 
should check probabilistic feasibility of a candidate solution, and (2) an Update Rule, which exploits the 
convexity of the problem for constructing a new candidate solution based on the Oracle outcome. All the 
available algorithms can hence be recast in the form of the following meta-algorithm. 
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1. Initialization: Set k = 0 and choose an initial candidate solution 6. 

2. Oracle: Invoke the Oracle with 0;,. The Oracle returns true if 4, is a probabilistic robust design. In 
this case, Exit returning 9seq = 9. Otherwise, the Oracle returns false, together with a violation 
certificate qx, that is a realization of the uncertainty q such that f (0;, qx) > 0. 

3. Update: Construct the new candidate solution 6;,) based on 6; and on the violation certificate qx. 
Outer iteration: Set k = k + 1 and Goto 2. 


In the next sections, we describe in detail the two basic components of the previous algorithm: the 
Oracle and the Update Rule. 


65.3.1 Probabilistic Oracle 


The Oracle constitutes the randomized part of the algorithm, and its role is to decide on the probabilistic 
feasibility of the current solution. This decision is made based on random samples of the uncertainty. 
More precisely, a number Nx of independently identically distributed (iid) uncertainty samples 


g,....q® EQ, 


are drawn according to the underlying distribution Pr, and the candidate design 0; is deemed probabilis- 
tically robust if 
£(0x.g) $0, 1=1,2,...,Ne. 


This leads to the following simple randomized scheme. 


Algorithm 65.1: Oracle 


Input: 0%,Nx 
Output: feas (true/false) and violation certificate qx 


for i=0to N;, do 
draw a random sample q'? 


Randomized Test 


if f(q, 0) > 0 then 
set qx = q, feas=false 
exit and return q;, 
end if 
end for 
set feas=true 


Note that at step k the feasibility of the candidate solution 0; is verified with respect to a number of 
samples Nx. If the test is passed, the solution is deemed feasible; otherwise, the uncertainty value q, for 
which the randomized test failed is returned as a violation certificate. The sample size N, depends on k, 
and has to be chosen to guarantee the desired probabilistic properties of the solution. Before discussing 
this issue in Section 65.3.3, we concentrate our attention on the outer loop of the algorithm, and show 
how one can update the current solution if the Oracle establishes its unfeasibility. 


65.3.2 Update Rules 


Various update rules have been proposed in the literature on randomized algorithms. For clarity of 
presentation, we discuss in detail only the simplest one, which is based on a (sub)gradient descent 
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technique. In this approach, it is assumed that a subgradient 0),(0) of the function f (9, q) at the violation 
certificate q, is computable. In the case when f (9, qx) is differentiable at 6, then 0; (8) is simply the gradient 
of f, that is, 


x (8) = Vof (9 4x). 
Then, the Update Rule consists of a classical gradient descent step. This is summarized in Algorithm 65.2. 


The main distinguishing feature of the method lies in the particular choice of the stepsize n;, which is 
given by 


F (9k, 8x) : 
——__+4r if O(0,) £0, 
Ne = ¥ WOx(x)II es (65.8) 
0 otherwise, 


where r is a positive parameter. 


Algorithm 65.2: UpdateRu1e (gradient method) 


Input: 9%, 4k 
Output: 0x41 


compute the subgradient 0;(0) of f (9, qx). 
compute the stepsize yn, according to (Equation 65.8). 


set 
OK (8x) . 
0; —Ne————__ if 0, (0 0 
Oi=y- TaD KO) A 
() 


k otherwise. 


Remark 65.1: Subgradient for LMIs 


Note that, in the case of the LMI defined in Equation 65.6, a subgradient of the function f(9, qx) = 
max (F(0, qx)) at 9 = 0% is readily computable as 


+ 
OK (9K) = [émax Fi (qk) Emax aS Ena Fis (4x) 8max > 


where Emax is a unit norm eigenvector associated with the largest eigenvalue of F(9;, qx). 

The specific choice of stepsize in Equation 65.8, and in particular the selection of the parameter r, 
has the purpose of guaranteeing finite-time convergence, by imposing a termination condition to the 
sequential algorithm when the oracle cannot find any probabilistic solution. Note in fact that, in this 
situation, the meta-algorithm introduced in the previous section would run indefinitely. In the next 
section, we show how to avoid this behavior, by imposing a finite termination condition to the algorithm. 
Then, we analyze the probabilistic properties of the ensuing randomized algorithm. 


65.3.3 Probabilistic Properties 


Let us first define the quantity 
R2 
Nouter = =| > (65.9) 


where R is the distance of the initial solution 6) from the center of a ball of radius r contained in the 
solution set S. 

To study the probabilistic properties of the iterative scheme discussed in this section, we fix desired 
probabilistic levels €, 8 € (0, 1) and assume that at step k the Oracle is invoked with the sample size which 
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has been derived in [8] 
m7 (k + 1)? 
log ————— 


Nx = Noracte(€, 8, k) = 2 : (65.10) 
log 


l-e 


With these positions, the meta-algorithm can be formally rewritten as follows. 


Algorithm 65.3: SequentialDesign 


Input: €,8 € (0,1), Nouter 
Output: 9seq 


Initialization 
choose 69 € R™, setk = 0 and feas=false 
Outer Iteration 
while feas=falseandk < Nouter do 
determine the sample size N; according to Equation 65.10 
Oracle 
(feas, qx) = Oracle(O;, Nz) 
Update 
if feas=false then 
update 0,4); = UpdateRule(0,;) 
else 
set Oseq = OK 
end if 


setk=k+1 
end while 


The probabilistic properties of Algorithm 65.3 are formally stated in the next theorem. 


Theorem 65.1: Probabilistic Properties 


Let the Convexity Assumption hold and let ¢,8 € (0,1) be given probability levels. Then, the following 
statements hold: 


1. The probability that Algorithm 65.3 terminates at some outer iteration k < Nouter and returns 9seq 
such that V(0seq) > € is less than 8. 
2. If Algorithm 65.3 reaches the outer iteration Nouter, then the problem is not r-feasible. 


Case 1 corresponds to a successful exit of the algorithm: the algorithm returned, with high probability, 
a probabilistic robust design seq. Case 2 represents instead an unsuccessful exit: no solution has been 
found. In this situation, however, we have a certificate q, which shows that the original problem is not 
r-feasible. We remark that, to determine Nouter, the quantities r and R have to be available. In particular, 
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r can be chosen to represent the desired level of r-feasibility, according to Definition 65.1. The radius R 
in Equation 65.9 can be instead replaced by an appropriate upper bound which can be easily estimated a 
priori. 

We also remark that the sample size N; provided in Equation 65.10 is independent of the number of 
uncertain and design parameters. This represents one of the key features of randomized algorithms: in 
general, the number of required computations is independent of the problem dimension, and therefore 
these algorithms are polynomial-time. For this reason, they are said to break the curse of dimensionality 
at the expense of a probability of violation; see [11]. 


65.3.4 Advanced Techniques 


The update rule discussed in the previous section is one of the simplest methods that are currently avail- 
able. More sophisticated techniques that still guarantee probabilistic properties of the ensuing solution 
can be developed. With these techniques, improved convergence rates are provided. 

In particular, different techniques have been proposed falling in the class of the so-called localization 
methods. In these methods, the update rule is based on the computation, at each step k, of a center of a 
suitably defined localization set £,. The set £; is guaranteed to contain the feasible set S, that is, S C Ly. 
The set £; is constructed based on the violation certificate q, returned by the Oracle. The point q; is used 
to construct a separating hyperplane hy = {E € R™ : ap = bx} having the property that 


a, OK > by and a, 0 <b, WOES. 


The separating hyperplane h, indicates that the half-space {0 : a, 8 > by} cannot contain a feasible point 
and can therefore be eliminated (cut) in subsequent steps of the algorithm. In this case, we know that 
SCLEN Hg, where 

Hy = (0: ap 0 < by}, 


and the algorithm constructs an updated localization set £,41 such that 
Levi 2 Le OHk. 


A new query point 0,41 € £441 is then computed, and the process is repeated. This is summarized in the 
following scheme. 


Algorithm 65.4: UpdateRul1e (localization methods) 


Input: 9k, x 
Output: 0x41 


compute the subgradient 0;(0) of f (8, qx) 
construct the half-space H;, based on the subgradient 
update the localization set Ly41 D LEN Hk 


return 0,4) = Center(L,) 


Different methods descend from different choices of the shape and description of the localization 
set. In particular, in probabilistic cutting plane methods, the localization set is a polytope, and the 
candidate solution 0;,; is a center of this polytope (usually, the analytic center). In the probabilistic 
ellipsoid algorithm, the localization set is instead an ellipsoid and the candidate solution is the ellipsoid 
center. 

It should be noted that Theorem 65.1 still holds also for the update rules introduced in this sec- 
tion, provided that Nouter is properly chosen. In particular, for the ellipsoid methods Nouter grows as 
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O(ng log(./noR/r)), while for the best-known cutting-plane method it is of the order of O(ng log” (R/r)), 
where in the first case R represents the radius of a ball and in the second case the radius of a cube, both 
inscribing the set S, and r is the desired r-feasibility level. 


Example 65.2: Probabilistic Sequential Design 


To show how the sequential algorithm presented in this section can be applied to a specific design 
problem, we revisit the aircraft lateral motion design example. In particular, we set a = 0.5 and seek 
a probabilistic feasible solution 6 = {P, W} to the uncertain LMIs 


P> Bl, (65.11) 
A(q)P + PA' (q) + B(q)W! + WB! (q) + 2aP <0, (65.12) 


where the uncertainty q is assumed to vary in the set Qp defined in Equation 65.2 and B = 0.01. We 
apply the algorithm SequentialDesign with ellipsoid update rule, and probability levels « = 0.01, 
8 = 10-°®. With this setting, the algorithm is guaranteed to return (with 99.9999% probability) a 
solution P, W such that Equations 65.11 and 65.12 hold with 99% probability. 

The algorithm was run with random initial candidate solution 99, and terminated after k = 28 
outer iterations returning the solution 


- 03075 -—0.3164 -—0.0973 —0.0188 

p.. — | 703164 05822 —0.0703 —0.0993 65.13) 

seq ~ | -0.0973 —0.0703 0.2277 0.2661 |’ ; 

| -0.0188 -—0.0993 0.2661 0.7100 
F—0.0191 0.2733 | 
—0.0920 0.4325 

Wseq = 0.0803 —0.3821 |" e544) 
| 0.4496 —0.2032 


This solution was deemed probabilistically feasible by the Oracle after checking Equations 65.11 
and 65.12 for Ny, = 2,089 uncertainty samples. Then, the probabilistic controller is constructed as 


ke. =| 729781 —1.9139  —3.2831 1.5169 
seq | 73922 5.1010 4.1401 —0.9284]° 


Further properties of this controller (worst-case and probabilistic) are studied in Section 65.6. 


65.4 Scenario Approach to Optimization Problems 


The sequential methods presented in the previous sections have been developed for feasibility problems. In 
this section, we present a nonsequential (batch) method for addressing in a probabilistic setting uncertain 
convex optimization problems, that is, optimization programs of the form 


min c'@ subject to f(0,q) <0, forall qe Q,. (65.15) 


6eR"0 


A probabilistic solution to this problem is found by replacing the semiinfinite set of constraints f (0, q) < 
0 (one for every possible value of q € Q,) with a finite number N of constraints f (0, q) < 0,i=1,...,N, 
one for each random sample q“). This approach is summarized in the next algorithm. 
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Algorithm 65.5: ScenarioDesign 


Input: €,8, no 
Output: 9Oscen 


compute the sample size N > Ngcen(€, 8, 19) as in Equation 65.18 
draw N iid. samples q™,..., gq 


Scenario Problem 


solve the convex optimization problem 
Oscen = arg min c'0 subject to f (0, q?) <0, i=1,...,N (65.16) 
6eR" 
To analyze the properties of the proposed algorithm, we consider for simplicity the case when the 


scenario problem (Equation 65.16) admits a feasible solution Ocen and that this solution is unique.* We 
state here a result that was originally presented in [1], see also subsequent improvements in [26]. 


Theorem 65.2: Convex Scenario Design 


Let the Convexity Assumption hold. Suppose that N > no, and «,8 € (0, 1) satisfy the inequality 


("Ja — No < 8, (65.17) 
(3) 


then the probability that V (8scen) > € is at most 8. 


The bound in Equation 65.17 provides an implicit relation between N, ¢, , and ng. This relation can 
be made explicit to derive the sample complexity of the scenario approach. We now state a bound derived 
in [14], proving that the sample complexity is proportional to 1/e for fixed 8. In particular, it was shown 
that, for given «, 8 € (0, 1), Equation 65.17 holds if 


2 1 2 
NS Nols days E In = +21 + ett | (65.18) 
€ € 


Example 65.3: Scenario Design 


To exemplify the sample complexity involved in a scenario design, we consider the problem of 
determining a probabilistic solution 6 = {P, W} of the optimization problem 


ally TrP subject to Equations 65.11 and 65.12 


, 


where Tr denotes the trace of a matrix. 


Note that, if the scenario problem (Equation 65.16) is unfeasible, then clearly also the original robust convex program 
(Equation 65.15) is unfeasible. The assumption on uniqueness of the solution can be relaxed in most cases, as shown in 
Appendix A of [1]. 
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In this case, if we set as previously « = 0.01, 8 = 10-6, applying the bound (Equation 65.18) with 
Ng = 18, we obtain Nscen = 7,652. This means that we have to solve an optimization problem with 
7,652 LMI constraints and 18 design variables. Then, Algorithm 65.5 returned the solution 


[ 0.1445 —0.0728 0.0035 0.0085 
Pecan = | 70.0728 0.2192 —0.0078 nae 
0.0035 -0.0078 0.1375 0.0604 |’ 

| 0.0085 —0.0174 0.0604 aia 


[0.0109 0.0908 
7.2929 3.4846 
0.0439 —0.0565 
[0.6087 —3.9182 


Wscen — 


Then, a probabilistic controller is constructed as 


20.0816 40.3852 —0.4946 ace 


Kseen = [So 70a1 18.1058 9.8937 —21.7363 


In the next section, we consider the more general case when the function f(9, q) may be nonconvex 
in 0. 


65.5 Learning Approach to Nonconvex Optimization 


A nonsequential probabilistic approach for nonconvex control design is based on statistical learning 
theory; see [13] for further details. One of the objectives of this theory is to derive uniform convergence 
laws. Hence, from the control point of view, the main utility of statistical learning is to derive convergence 
results and compute the sample complexity which hold uniformly for all controller parameters 0. In turn, 
this leads to a powerful methodology for control synthesis which is not based on a convexity assumption 
on the controller parameters. The obtained sample complexity bounds are significantly larger than those 
derived in the convex case, even though the structure of the bounds show various similarities. We also 
remark that the setting of statistical learning may be seen as a major extension of the classical Monte 
Carlo approach which is useful only in the context of a posteriori analysis (see Section 65.6) where the 
controller parameters are fixed. 
To treat the problem in full generality, we consider the following optimization problem 


min J(®@) subject to g(0,q)=0, forall qeQ), (65.19) 
ER" 


where the cost function J : R”’ — R and the binary function g : R™ x Q, — {0,1} may be nonconvex. 
Note that constraints of the form (Equation 65.5) may be recast in this framework by setting 


(8,4) =1(F (4) > 9), 


where the indicator function I(-) is one if the clause is true and zero otherwise. We consider here only the 
case when the binary function g(, q) can be written as a Boolean polynomial expression on the decision 
variable 9, as now formally stated. 


Definition 65.2: (y, m)-Boolean Function 


The function g:R™ x Qp — {0,1} is a (y,m)-Boolean if, for fixed q, it can be written as a Boolean 
expression consisting of Boolean operators involving m polynomials 


B1(9),. . . »Bm(9) 
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in the variables 0;,i=1,...,n9, and the degree with respect to 0; of all these polynomials is no larger than 
y>0. 


A probabilistic approximate solution to problem (Equation 65.19) may be found by means of the 
following algorithm, which has a similar structure of Algorithm 65.5. 


Algorithm 65.6: NonConvex 


Input: ¢,8, no 
Output: ®8ncon 


compute the sample size N > Nnycon(€, 8, 19) as in Equation 65.21 
draw N iid. samples q™,...,q) 


Nonconvex Scenario Problem 
solve the nonconvex optimization problem 


Oncon = arg qin J(8) subject to g(0,q) =0, i=1,...,N (65.20) 


Note that Equation 65.20 requires in general the solution of a nonconvex optimization problem. This 
usually leads only to a local minimum. However, the following theorem, stated in [14], guarantees that 
this minimum is €-feasible. 


Theorem 65.3: Nonconvex Learning-Based Design 


Let g(0, q) be (y, m)-Boolean. Given € € (0, 0.14) and 8 € (0, 1), if 


4.1 (. 21.64 8 
N > Noucon(€8, tt) = | (in —— +4399 log, ( my) (65.21) 
€ € 


where e is the Euler number, then the probability that V(®ncon) > € is less than 8. 


Example 65.4: Nonconvex Learning-Based Design 


In contrast with the previous examples which considered quadratic stability, we aim here at designing 
acontroller K that minimizes (in probability) the decay rate a of Hurwitz stability. Moreover, as in [18], 
we want to impose a saturation constraint on the entries of the gain matrix K. That is, we aim at 
solving the following nonconvex optimization problem 


min (—a) 
a,K 
subject to 
A(q) + B(q)K + a is Hurwitz for all g € Qp (65.22) 
—Kip<Kij<Kij §=1,2, f=1,2,3,4 (65.23) 
where 
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To use the framework developed in this section, we set the design parameters to 0 = {a, K}. 
Then, we note that checking the constraint in Equation 65.22 can be performed using the classical 
Hurwitz test, which requires strict positivity of all the Hurwitz determinants H;(6,q), i= 1,...,n. That 
is, Equation 65.22 is equivalent to the Boolean condition 


Bu(8,q) = {H1(8,q) > 0} A --+ A {Hn,(8,q) > O}, 


where the symbol A stands for “and.” Note now that H;(9,q) are ns polynomials in 8, where ns; = 4 
is the dimension of the state matrix, whose degree is at most yj = i(i+ 1)/2; see [12] for additional 
details. Hence, By(6, q) is (y, ns)-Boolean, with 


y=. max yj =Ns(ns + 1)/2 = 10. 


i=1,....Ns5 
Additionally, constraint (65.23) is immediately rewritten as the Boolean condition 
Bx (9) = {K,1 = —Kia} A {Kt1 < Kaa} A ++ A {Koa = —Kaa} A (Koa < Koa} 
which is (1, 16)-Boolean. Hence, the binary function 


. JO if By(8,q) A Bx (9), 
9(8,q) = otherwise, 


is (10, 20)-Boolean. Then, we let « = 0.01 and 8 = 10~§, and, noting that ng = 9, y = 10, m = 20, we 
apply Theorem 65.3 obtaining Nncon(é, 8, N9) = 310, 341. A (local) solution of the ensuing nonconvex 
scenario problem (Equation 65.20) was computed by means of a numerical optimization method 
based on local linearization obtaining 


Qncon = 3.7285, 


0.8622 02714 —5.0000 aca 


Kncon = [3 5000 1.4299 3.9328 1.0000 


We note that three gains in the matrix Kycon saturate to their extreme values. 


65.6 A Posteriori Performance Analysis 


In this section, we study the performance of the probabilistic controller selected with the sequential 
or nonsequential methods previously presented. Since the controller is fixed, this amounts to solving 
a-posteriori deterministic and probabilistic analysis problems, as discussed next. 


65.6.1 Deterministic Analysis 


For a fixed value of the design parameter 0, the deterministic, or worst-case, analysis consists in computing 
the radius of deterministic performance pwc, which is the largest value of p € [0, 1] for which the constraint 


f(9,q) <0 


is robustly satisfied for all q in the set Qp. Then, once this worst-case analysis is completed, the objective 
is to perform a probabilistic analysis beyond the radius of deterministic performance, that is, for values 
of p € [Pwe, 1]. 

Deterministic and probabilistic analysis are now illustrated more precisely for the example con- 
sidered in this chapter. First, we perform a worst-case analysis of the controller Oeq derived in 
Section 65.3. 
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Example 65.5: Deterministic Analysis 


Note that in the example under consideration the entries of 


0 1 0 0 0 0 
Oo q2 43 0) q10 

A(q) = , BqQ= 
‘ qa COO 95 —1 4 q11 0 
94946 97 98+49596 4o- 46 MN2+96911 13 


depend multiaffinely* on the uncertainty gq. In this case, for fixed p, to detect quadratic performance 
of an uncertain system affected by multiaffine uncertainty, it is well-known (see, e.g., [20]), that it 
suffices to check the simultaneous satisfaction of the uncertain constraint (Equation 65.3) at the 
vertices of the hyperrectangle Qp. That is, for given P > 0 and W, if the inequality 


p(P, W,v!) = A(v')P + PAT (Vv!) +. B(v/)W T+ v!) + 2aP < 5 24 
Dop(P, W,v!) = A(v')P + PA! (v') + B(v')W! + WB! (v') + 2aP < 0 (65 24) 


is satisfied for all vertices v',i =1,2,.. 20 of the hyperrectangle Qp, then the uncertain constraint 
(Equation 65.3) is satisfied for all q € Qp. 

Then, computing the radius pwc amounts to solving a one-dimensional problem in the variable 
p and, for each value of p, to verify if the inequality (Equation 65.24) is satisfied for all vertices of 
Qp. This problem can be solved, for example, using a bisection method, but for each value of p an 
exponential number of vertices 2¢ should be considered. 

Performing this worst-case analysis on the design Pseq and Wseq derived in Equations 65.13 and 
65.14, we compute 

Pwe © 0.12. 


We conclude that the controller derived for the aircraft model is robustly stable and attains robust 
quadratic performance for all values of q € Qp, with p € [0, pwc]. 


65.6.2 Probabilistic Analysis 


In this case, the a-posteriori analysis consists of designing a Monte Carlo experiment which is based 
on random extractions of uncertainty samples. The result of the experiment is to return an estimated 
probability of satisfaction of the uncertain constraint under attention. The probability provided by this 
randomized algorithm is within an a-priori accuracy € € (0, 1) from the true probability, with confidence 
8 € (0, 1). That is, the algorithm may indeed fail to return an approximately correct estimate, but the 
probability of failure is at most 8. 

More precisely, let g € Qp represents the random uncertainty acting on the system. Then, for a given 
value of the design parameter 0, we aim at providing an estimate Ry of the probability of performance 


R(0) = Pr {q EQ, : f(8,q) < o} 


using N iid. samples q®,i=1,2,...,N of q € Qo. The estimate Ry, called the empirical probability, is 
given by 


N 

A 1 . 

Rv = 5 S>1f (4) < 0. 
i=1 


The sample complexity N is related to the accuracy and confidence and it can be determined using fairly 
classical large deviations inequalities. In particular, the so-called Chernoff Bound [5] can be readily used 


* A function f : R¢ > R is said to be multiaffine if the following condition holds: If all components q1,...,qg except one 
are fixed, then f is affine. For example, f(q) = 3q1q2q43 — 69193 + 44293 + 2q1 — 2q2 + q3 — 1 is multiaffine. 
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to determine the number of required samples. More precisely, given €, 8 € (0, 1), if 


1 2 
N>N )=}|—In<], 
> Ncher(€, 8) Ee ne 
then, the probability inequality 
|R(®) — Ry| <€ 


holds with probability at least 1— 8. We remark that the sample size Ncher provided by the Chernoff 
Bound is independent of the number @ of uncertain parameters q. 
The following algorithm describes the Monte Carlo experiment discussed so far. 


Algorithm 65.7: Probabilistic Analysis 
Input: €,8,9 
Output: Rn 


compute the sample size N > Ncher(€, 8) 
draw N iid. samples q,...,q 
return Ry = N eae ILf (0, 4) < 0] 


Then, the next step is to construct the probability degradation function, which is the plot of the 
probability of stability as a function of the radius p. This plot may be compared with the radius of 
performance pwc, to provide additional important information to the control engineer on the behavior 
of his/her design beyond the worst-case performance margin. This is illustrated in the next example. 


Example 65.6: Probabilistic Analysis 


For the controller seq, we proceed with a probabilistic analysis for perturbations q whose radius goes 
beyond the deterministic radius pwc previously computed. That is, we study how the probability of 
quadratic performance degrades with increasing radius p. More precisely, for fixed p > pwc, we 
compute an estimate Ry. Note that this analysis can be carried out using smaller values of «, 8 than 
those employed in the design phase. For instance, taking « = 0.005, 8= 10~©, by means of the 
Chernoff bound we obtain N = 290, 174. Then, we estimated the probability degradation function 
for 100 equispaced values of p in the range [0.12, 0.5]. For each grid point the estimated probability 
of performance is computed by means of Algorithm 65.7. For each value of p, the accuracy of this 
estimate satisfies the inequality ‘ 
IR(8seq) — Rnl <€ 

with probability at least 1 — 5. 

The obtained results showing the estimated probability together with the deterministic radius 
Pwc are given in Figure 65.1. From this plot we observe, for instance, that if a 2% loss of probabilistic 
performance may be tolerated, then the performance margin may be increased by approximately 
270% with respect to its deterministic counterpart. In fact, for p = 0.34, the estimated probability of 
performance is 0.98. In addition, we note that the estimated probability is equal to one for values of 
the radius up to p ¥ 0 26. 

In Figure 65 2, we plot the closed-loop eigenvalues for p = 0.34. 


65.7 Randomized Algorithms Control Toolbox 


The algorithms discussed in this chapter may be readily implemented in MATLAB®. However, while 
this may be straightforward in the case of analysis problems, it may require a nontrivial effort for the 
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Probability degradation function 
i) 
© 
T 
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FIGURE 65.1 Probability degradation function. 


design problems discussed in Sections 65.3 through 65.5. For this reason, to render these techniques easily 
accessible to the interested researchers, an effort was made to unify them into a coherent set of MATLAB 
routines. This effort led to the release of a Randomized Algorithms Control Toolbox (RACT); see [36]. 
This toolbox provides convenient uncertain object manipulation and implementation of randomized 
methods using state-of-the-art theoretical and algorithmic results. The two main features of the package 


2.5 T T T T 


Imag 
°o 


FIGURE 65.2 Closed-loop eigenvalues for p = 0.34. 
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are a functional approach with m-file templates, and a definition of design problems in generic LMI format 
using the widely used YALMIP syntax. This first release of the toolbox provides an easy-to-use interface 
of current randomized algorithms for control and is intended to be used by researchers, engineers, and 
students interested in robust control, uncertain systems, and optimization. The package can be freely 
downloaded from http://ract.sourceforge.net. 


RACT features include 


¢ Definition of a variety of uncertain objects: scalar, vector, and matrix uncertainties, with different 
density functions. 

¢ Easy and fast sampling of uncertain objects of almost any type. 

¢ Randomized algorithms for probabilistic performance verification and probabilistic worst-case 
performance. 

¢ Randomized algorithms for feasibility of uncertain LMIs using stochastic gradient, ellipsoid, or 
cutting plane methods. 

¢ Optimal design methods using scenario approach. 


65.8 Miscellaneous Topics 


65.8.1 General Uncertainty Description 


In this chapter, we have studied the case when the uncertain parameters q are bounded in a hyperrectangle 
Qp. However, in some applications, different bounding sets, such as ellipsoids or diamonds; see, for 
example, [20], are more appropriate and should be studied. In other situations, the uncertainty affecting 
the system is of general type, that is, it consists of parametric and nonparametric uncertainty. In this 
case, various performance problems may be reformulated using the so-called M — A configuration and 
the related \1-theory, see [11], which is a quite flexible and general representation of linear uncertain 
systems. From the probabilistic point of view, structured uncertainty descriptions require to develop 
suitable randomized algorithms for generation of (vector and matrix) uncertainty samples within various 
bounding sets, see Section 65.8.2 for additional details. 

Moreover, in the probabilistic setting described in this chapter, the probability measure Pr of the 
uncertainty q is assumed to be known. If this is not the case, then clearly the probability of violation 
depends on the specific choice of the measure Pr. The result is that, in extreme cases, the probability 
of violation may vary between zero and one when taking different measures. Therefore, without any 
guideline on the choice of the measure Pr, the obtained probability of violation, or its estimate, may be 
meaningless. In various applications, the probability measure may be estimated directly from available 
data, but in general the selection of the “right” measure should be performed with great care. This problem 
has been studied in [19,21]. In particular, in [21], the theory of distribution-free robustness, which has the 
objective to determine the worst-case measure in a given class of bell-shaped distributions, is discussed. 


65.8.2 Sample Generation Algorithms 


All the previously described randomized methods critically rely on efficient techniques for random sample 
generation. The interested reader may refer to [11] for a general discussion on the topic. In this section, 
we briefly discuss the problem of generating uniform i.i.d. samples in the 4, norm-ball of radius p 


BE = {qeER": lallp < 0}. 


In particular, we report an algorithm, presented in [23], that returns a real random vector q uniformly 
distributed in the norm-ball Be. This algorithm is based on the Generalized Gamma density Ga (x), 
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defined as 


where a and ¢ are given parameters and I'(a) is the Gamma function. 


Algorithm 65.8: Uniform Generation 


Input: 0,p 
Output: q uniformly distributed in Be 


generate ¢ independent random real scalars €; ~ Gj [pp 
construct the vector x € R¢ of components x; = sj&;, where s; are i.i.d. random signs 


generate z = w'/£, where w is uniform in [0, 1] 
te 


return q = pz IxIp 


Figure 65.3 visualizes the main steps of this algorithm in the simple case of sample generation of 
two-dimensional real vectors in a circle of radius one (¢ = 2, p = 2, p = 1). First, we note that for p = 2 
the Generalized Gamma density Gi p*) is related to the Gaussian density function. The random samples 


drawn from a Gaussian distribution (step 1 in the figure) are radially symmetric with respect to the é2 
norm. Roughly speaking, this means that their level curves are ¢2-spheres. Secondly, the samples are 
normalized obtaining random vectors uniformly distributed on the boundary of the circle (step 2), and 
then injected according to the volumetric factor z (step 3). 

We remark that in [11] a similar algorithm for complex vectors is presented. The sample generation 
problem becomes much harder when we are interested in the uniform generation of real and complex 
matrix samples A bounded in the induced ¢,-norm. Specific algorithms are presented in [11]. 


65.8.3 Mixed Deterministic and Probabilistic Setting 


We remark that the setup considered here can be easily extended to mixed deterministic and random 
uncertainties. To this end, consider our motivating example. In this example, we treated all the parameters 
as random ones, but we could easily deal with the case when we want to robustly guarantee performance 
against some of them. 

Just for exemplification, assume for instance that the vector q is partitioned as q = {4prob» qaet}, so that 


the first m uncertain parameters qprob = [a1 tee Gr have probabilistic nature with uniform distribution 


in the hyperrectangle Q,,op, and the parameters qdet = [Qm+1 tee qe|' are deterministic uncertainties, 
unknown but bounded in the hyperrectangle Qge, where the sets Qprob, Qaet are defined accordingly. 
In this case, the present approach can be extended considering the following set of LMIs which depend 


on q prob 


A(4prob> v')P “P PA' (Gprob> v!) =e B(Gprob> vi ) wl aE wB! (prob v!) +2aP <0 
(65.25) 
A(Qprob> vM)p + PA! (dprob> yM) + B(Gprob wl + ws! (Gprob> yi) + 2a0P <0 
where v', i= 1,...,M = 2°" are the vertices of the hyperrectangle Qget. 
Then, an €-feasible solution to the uncertain constraint (Equation 65.25) can be found using the tools 


described in Sections 65.2 through 65.4. Note that this solution would guarantee, as desired, robust 
performance with respect to qaet and probabilistic performance with respect to dprob- 
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FIGURE 65.3 Generation of random vectors uniformly distributed in a circle. 


65.8.4 Linear Parameter-Varying Systems 


LPVs provide a good starting point for control design of fairly general classes of gain scheduling prob- 
lems. This methodology is suitable and frequently used, for example, in aerospace control and in other 
applications as well, due to the time variations of the operating conditions. In this context, the system 
matrices depend on an unknown scheduling parameter vector &(t) which can be measured online by the 
controller at each time instant t. More precisely, the LPV system is described by the equations 


X(t) = A(E(t))x(t) + BEE (E))u(e), 
y(t) = C(E(A))u(t) 
and the feedback controller is of the form 
u(t) = K(&(t))y(E). 


Since the original motivation for introducing gain scheduling is to handle plant nonlinearities, a critical 
issue of the resulting LPV model is to properly deal with the time-varying parameters &(¢). An LPV 
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performance problem can be easily reformulated using parameter-dependent LMIs, but these inequalities 
are nonlinear with respect to & and it is practically impossible to solve them exactly. To this end, three 
different approaches have been developed in recent years and are now described. 

The first approach (approximation LPV) studies a specific class of functions of the scheduling param- 
eters. For example, the entries of the matrices of the LPV model can be taken as multiaffine functions, or 
linear fractional transformations, of &. This technique reduces the design problem to tractable formulae 
which involve a finite number of LMIs, but some conservatism is introduced in the approximation. 

The second method (gridding LPV) requires to grid the parameters & which are assumed to be bounded 
within a given set &. In this case, control design requires the solution of a finite set of LMIs. However, the 
number of LMIs depends on the grid points and it exponentially increases with the number of scheduling 
parameters. In addition, the satisfaction of the LMI conditions at the grid points does not provide any 
guarantee of their satisfaction for the entire set ©. 

In the third technique (probabilistic LPV), the scheduling parameters & entering into the LMI equations 
are treated as random variables. Therefore, to compute a probabilistic controller, a randomized sequential 
algorithm of the form of Algorithm 65.3 can be derived. Clearly, in contrast to the previous deterministic 
methods, the obtained results enjoy probabilistic properties similar to those stated in Theorem 65.1. In [6], 
a specific technique for LPV systems, based on the simultaneous update of two gradient iterations similar 
to those in Algorithm 65.2, is developed. We refer to this chapter for further details and specific numerical 
results for feedback design of the lateral motion of an aircraft with nine scheduling parameters. 


65.8.5 Systems and Control Applications 


Several applications of probabilistic and randomized methods have been studied. In particular, we recall 
the following: 


« Aerospace control: Applications of randomized strategies for the design of control algorithms in 
the field of aeronautics and aerospace was initiated by Stengel (see, e.g., [34]). In [32], a modern 
approach based on ellipsoid techniques is proposed for the design of an LPV control of an F-16 
aircraft. 

¢ Flexible and truss structures: Probabilistic robustness of flexible structures consisting of a mass- 
spring-damper model affected by random bounded uncertainty with force actuators and position 
sensors. Comparisons with standard robustness techniques are made [2]. In the field of truss 
topology optimization, a scenario-based approach was proposed in [24]. 

¢ Model (in)validation: A computationally efficient algorithm for model (in)validation in the presence 
of structured uncertainty and robust performance over finite horizons was proposed in [35]. 

¢ Adaptive control: A methodology for the design of cautious adaptive controllers based on a two-step 
procedure is introduced in [25]. First, a probability measure is updated online based on observations; 
then a controller with certain robust control specifications is tuned to this updated probability by 
means of randomized algorithms. 

¢ Switched systems: Randomized algorithms for synthesis of multimodal systems with state-dependent 
switching rules. Convergence properties (with probability one) of nonconvex sequential methods 
are analyzed. Simulations show the efficacy of the method for various practical problems [29]. 

¢ Network control: Congestion control of high-speed communication networks by means of random- 
ized algorithms. Various methods are developed and compared using different network topologies, 
including Monte Carlo and Quasi-Monte Carlo techniques [17]. 

« Automotive: A randomization-based methodological approach for validation of advanced driver 
assistance systems is studied in [28]. The case study also points out some characteristic properties 
of randomized algorithms regarding the necessary sample complexity, and the sensitivity to model 
uncertainty. 

¢ Model predictive control (MPC), fault detection, and isolation (FDI): Sequential methods (ellipsoid- 
based) are derived in [30] to design robustly stable finite-horizon MPC schemes for linear uncertain 
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systems, when the uncertainty is not restricted to some specific class. In [33], a risk-adjusted 
approach based on randomization is proposed for robust simultaneous fault detection and isolation 
of MIMO systems. 

¢ Circuits and embedded systems: Performance of electric circuits subject to uncertain components 
introduced by the manufacturing process. The objective is to evaluate the probability that a given 
“system property" holds providing “hard” (deterministic) bounds [31]. In [15,16], randomized tech- 
niques are applied to estimate the performance degradation of digital architectures and embedded 
systems subject to various sources of perturbation. 
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66.1 Introduction 


Stochastic differential equations of It6’s kind are a suitable model for the study of nonlinear dynamic 


systems subject to random disturbances, for example, marine vehicles in the presence of wave forcing. 
For It6 stochastic systems, Khas’minskii [8] and others have developed Lyapunov techniques for stability 
analysis of these systems. Since the mid-1990s this class of systems has been a fertile ground for feedback 
design, particularly for stabilization [1,2], stochastic disturbance attenuation [3], and adaptive control [3]. 

In this chapter we review various fundamental results in the area of control of nonlinear continuous- 
time stochastic systems. Our designs are presented through several worked examples. The emphasis is on 
the construction of feedback laws, without theorem statements. 
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66.2 Stability of Stochastic Systems 


We consider stochastic systems of the form 
dx =f (t,x) dt + h(t, x) dw, (66.1) 


where x € IR" is the state, x9 = x(0) is the initial state, w is an r-dimensional standard Brownian motion 
defined on the complete probability space (Q, F, {F;}:>0, P), with Q being a sample space, F being a 
o-field, {F;};>0 being a filtration, and P being a probability measure; f : IR; x IR" > IR" and h: Ry, x 
R"’ > R"™", 

For any given function V(t,x) € C!*(IR; x IR"; R;), associated with the stochastic differential equa- 
tion 66.1, we define the differential operator L as follows ( [8]): 


_ av, av 


LV 
ot . Ox 


me 7orV 
f4 ste {h an (66.2) 


Let t denote the maximal interval of existence of the solution process x(t) of the system (Equation 66.1). 
We say that the solution x(t) is regular if P{t = co} = 1. 

For system (Equation 66.1), suppose that f(t, x) and h(t, x) are locally Lipschitz and f(t, 0), h(t, 0) are 
bounded uniformly in f. If there exists a nonnegative function V(t, x) €C 12(R, x R";R,) such that for 
some c > 0, 


LV <cV, 
Ve= inf V(t,x) > co, asR>OO, 
lx||>R 


then system (Equation 66.1) has an almost surely unique (strong) solution on [0, 00). 


66.2.1 Boundedness in Probability 


A stochastic process {x(t), f > 0} is said to be bounded in probability if 


lim sup P{||x(£)|| >c}=0. 
CO 


f* 0<t<oo 


For system (Equation 66.1), if there exist functions V(t, x) € c!(Ry x R";R4), WiC), b2(-) € Koo such 
that j1(||x||) < V(t, x) < 2(I|xl]), and constants c > 0,c, > 0 such that CV < —c, V +c, then the solu- 
tion process is bounded in probability. 

Compared with deterministic systems, there is more than one way to define stability for stochastic sys- 
tems because in probability theory there are several concepts of convergence. Generally, these alternative 
stability definitions are not equivalent. Some common stability notions and (related Lyapunov-type tests) 
are given next. These basic notions are 


¢ Boundedness in probability (defined above) 

¢ Stochastic stability with “almost sure” convergence 

¢ Stability in probability 

¢ Noise-to-state stability (with respect to unknown but deterministic covariance) 
¢ Stochastic input-to-state stability (ISS) (with respect to a stochastic input) 

¢ Practical stochastic stability 
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66.2.2 Stability Notion 1 


For system (Equation 66.1) with f(t, 0) = 0, h(t, 0) = 0, the solution x(t) = 0 is said to be stochastically 
asymptotically stable in the large if for any € > 0, 


Ixol>o | iso 


lim P {= \|x(t)|| = | =0 


and for any initial condition xo, 
P{lim x(t)=0}=1. 
t>oo 
This form of stability, introduced in [8], is not an immediate analog of the standard deterministic 
stability concepts. In order to establish a clearer connection between deterministic stability results in the 


style of Hahn [6] or Khalil [7] and stochastic stability results, we introduce the following stability concepts 
from Krstic and Deng [9], which are based on class K functions [6] rather than on the € — 5 formalism. 


66.2.3 Stability Notion 2 
For system (Equation 66.1) with f(t, 0) = 0, h(t, 0) = 0, the solution x = 0 is 


¢ Globally stable in probability if for all « > 0 there exists a function y(-) € K such that 
P{ilx@Il < y(lxol)} 21-6 Vt>0, Wxo € IR"\{0}; 


¢ Globally asymptotically stable in probability if for all € > 0 there exists a function B(-, -) € KL ( [6]) 
such that 


P{ilx(2)ll < B(llxol,}>1—¢, Vt>0, Vxo € R"\{0}. 


For these stability notions, we have the following test. 


66.2.3.1 Test for Stability A 


Consider system (Equation 66.1) with f(t, 0) = 0, h(t, 0) = 0. If there exist functions V(t, x) € c!4(Ry x 
R”;R+), W1(-), W2(-) € Koo, and a continuous and nonnegative function W(x), such that 


Mi (Ilx|]) < Vit, x) < pe(lxll), 
LV <—W(x), 


then the equilibrium is globally stable in probability and P{lim;—.o0 W(x) = 0} = 1. Moreover, if W(x) 
is positive definite, the solution x = 0 is stochastically asymptotically stable in the large and also globally 
asymptotically stable in probability. 

The notion of deterministic ISS introduced by Sontag [15] plays an important role in nonlinear system 
analysis and synthesis. For stochastic nonlinear systems, the analogous ISS (see [14,16,17]) is a richer 
property as it can be considered relative to more than one class of inputs. 


66.2.4 Stability Notion 3 


Consider the system 
dx =f (t,x) dt + h(t, x) X(t) dw, (66.3) 


where & : IR; — R'*’ is Borel measurable and bounded, and X(t)X! (t) is the infinitesimal covariance 
function of driving noise X(t) dw. System (Equation 66.3) is said to be noise-to-state stable (NSS) if for 
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any € > 0, there exist functions B(-,-) € CL and y(-) € K, such that* 


O<s<t 


P{llx(t)Il < B(llxoll. t) +y ( sup 1210260") >1—6 Wt>0, Vxo € R"\{0}. 


66.2.4.1 Test for Stability B 


Consider system (Equation 66.3) and suppose there exist functions V(t,x) €C!?(Ry x R";R4), 
a1 (-), &2(-), e(-) € Koo, and a3(-) € K, such that 


ay ([lx|]) < V(t, x) < a2 ((IxIl); 


IIxl| = p(\|Z=7 Ip) > LV <—a3((Ixll). 


Then the system (Equation 66.1) is NSS. 
To introduce a different ISS notion, referred to as stochastic input-to-state stability, consider the 
following system: 


dx = f (x, v) dt + g(x, v) dw, (66.4) 


where v = v(x, tf): R” x Ry — R” is the input; w is an r-dimensional standard Brownian motion defined 
on the complete probability space (Q, F,{Ft}:>0, P), with Q being a sample space, F being a o-field, 
{Fi}1>0 being a filtration, and P being a probability measure; and f : R” x R” — R” andg: R” x R™ > 
R”**" are assumed to be locally Lipschitz in their arguments. Assume that for every initial condition xo 
and each essentially bounded measurable input v, system (Equation 66.4) has a unique solution x(t) on 
[0, 00) which is F;-adapted, t-continuous, and measurable with respect to B x F, where B denotes the 
Borel o-algebra of R. 


66.2.5 Stability Notion 4 


System (Equation 66.4) is practically stochastically input-to-state stable if for any given € > 0; there exist 
functions B(-,-) € KL, y(-) € K and a constant d > 0 such that 


Pisco < B(|lxollst) + ( sup i) +a > 1-6 VWt>0, VxoER"\{0}, (66.5) 
O<s<t 

where ||vs|| = inf 4ce,p()=0 sup{||v(x(a, 5), s)|| :@ € Q\A}. When d=0 in Equation 66.5, system 

(Equation 66.4) is said to be stochastically input-to-state stable (SISS). 

The property (Equation 66.5) ensures that when the input v = 0 and d = 0, system (Equation 66.4) is 
globally asymptotically stable in probability. It also ensures that the solution process of system (Equa- 
tion 66.4) is bounded in probability when the input v is bounded almost surely (in other words, there 
exists a constant M > 0 such that P{sup,s 9 || v(x(o, t), t)|| < M} = 1). When v(x, tf) = v(t) is deterministic 
and d = 0, the above definition is a generalization of NSS. 

AC? function V(x) is said to be a practical SISS-Lyapunov function for the system (Equation 66.4) if 
there exist Ko functions 0, &2, a, x, and a constant d > 0 such that 


oy ([]x[]) < V(x) < a(x), 
LV < x(Ivll) — a(||xll) +4. 


When d = 0, the function V is said to be an SISS-Lyapunov function for system (Equation 66.4). 


* For X € R"*", ||X||p presents the Frobenius form defined by ||X || p = / Tr(X!X). 
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66.2.5.1 Test for Stability C 


System (Equation 66.4) is practically SISS (resp. SISS) if there exists a practical SISS- (resp. SISS-) Lyapunov 
function. 
For other versions of “stochastic” ISS, refer to [16,17]. 


66.3 Stabilization of Stochastic Systems: The Basics 


Due to the It6 differentiation rule (Equation 66.2), additional quadratic terms, not encountered when 
working with deterministic systems, arise in the stochastic Lyapunov analysis. For this reason, a control law 
designed for a deterministic system does not necessarily result in a stabilizing design for the corresponding 
stochastic problem. Moreover, different notions of stability, different forms of Lyapunov functions, and 
different performance indices (in optimal control formulations) may be used in the stochastic case. Efforts 
toward Lyapunov stabilization of stochastic nonlinear systems started in the work of Florchinger [4,5] 
who suggested the use of control Lyapunov functions and Sontag’s stabilization formula in the stochastic 
setting. This was followed, by various other authors, by extensions of these concepts to problems with 
unknown noise covariance, adaptive control formulations, differential game and inverse optimal con- 
trol formulations, output-feedback designs, and numerous constructive recursive procedures employing 
“backstepping.” These efforts are reviewed here. 
Consider a stochastic system which, besides the noise input w, has a control input u: 


dx = f (x) dt + gi(x) dw + go(x)u dt, (66.6) 


where f (0) = 0, g1(0) =0, and u € IR”. We say that system (Equation 66.6) is globally asymptotically 
stabilizable in probability if there exists a control law u = a(x) continuous everywhere, with a(0) = 0, 
such that the equilibrium x = 0 of the closed-loop system is globally asymptotically stable in probability. 

A smooth positive definite radially unbounded function V : IR” > Ry, is called a stochastic control 
Lyapunov function (sclf) for system (Equation 66.6) if it satisfies 


1 gO V 
inf Lt ate 81 578! +1, Vu <0, Vx 40, 


ue 


where LyV= BV f(x), IgV= OV 99 (x). 
A positive definite radially unbounded function V(x) is an sclf if and only if for all x 4 0, 


1 pOrV 
Le.V=0 => LV t ie {« 78] <0. 

It is shown in [9] that system (Equation 66.6) is globally asymptotically stabilizable in probability if there 
exists an sclf with the small control property.* 

The efforts on constructive methods for stabilization of broad classes of stochastic nonlinear systems 
commenced with the result of Pan and Basar [11], who derived the first backstepping design for strict- 
feedback systems, employing a risk-sensitive cost criterion (this approach was revisited a decade later 
and extended in [13]). This result was immediately followed by extensive efforts by various authors 
employing Lyapunov techniques for stabilization of stochastic continuous-time systems. The key results 


* An sclf V(x) is said to satisfy the small control property if there exists a control law a¢(x) continuous on IR” such that 


1 av 
LV 4 sae ef a af +Lgy Vac <0, Wx £0. 


66-6 Control System Advanced Methods 


have focused on three classes of systems, which Pan [10] characterized as canonical forms and pre- 
sented coordinate-free differential geometric conditions for transforming a given stochastic system into 
a particular canonical form. 


1. Strict-feedback canonical form 


dx, = (x2 +41 (x1)) dt + Y- gia) dwi; 


i=1 


s (66.7) 
diy 1 = (Kn + dn1(X1 «6 %n—1)) dt + Y= Sin 11s». Xn—1) di, 
i=l 
dxy = (An(X1,..-5%n) + Onl, .-5%n)u) dt+ Y gin (x1 .--+Xn) dw. 
i=1 
2. Observer canonical form 
P s 
dx, = (» +ay(x1)+ be boos dt+ Y- g(x) dwi; 
i=1 i=1 
P Ss 
dx) = (» +ar(m)+ >> bal dt +) ° gir(x1) dwi, 
i=l i=1 
(66.8) 
P s 
dp = (os) +) > box dt +S ~ gin(x1) dwi, 
i=1 i=1 
yx}. 
3. Systems with zero dynamics 
dxz = az(Xz,x1) dt +) giz(xz) dwi, 
i=1 
dxy = (x2 + ai(xz,m1)) dt + D> gin (Xz, x1) dw, 
i=l 
(66.9) 


Ss 
Ap = (Kp Fy —1 (zs X15. Xp) dt + YD Bip aes H1y Xr) dws 


i=l 
Ss 
Ax, = (Ay (Xz,X1,...5Xp) + by (xz,X1,...,X~)u) dt + Yerba. .. 5X7) dwi, 
i=l 
yx}. 


In the next three sections we present control designs for the three classes of stochastic systems. 
The following design tools are illustrated in the subsequent sections: 
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¢ Stochastic backstepping (Section 66.4) 

¢ Observer-based output feedback for systems with stochastic forcing through nonlinear input vector 
fields, but with non-noisy output measurements (Section 66.5) 

« Domination of uncertain and unmeasured zero dynamics (Section 66.6) 

¢ Stochastic adaptive control subject to unknown noise covariance and other parametric uncertainties 
(Section 66.6) 

¢ Inverse optimal redesign leading to cost functionals given as expectations of time integrals of 
positive definite functions of the state and control (Section 66.7) 


66.4 Stabilization in Probability via Backstepping 


66.4.1 Introductory Example 


We introduce the design idea of stabilization in probability via backstepping for an example of a system 
in the strict-feedback canonical form. Consider the system 


dx, = x dt + gp! (x1) dw, 
pen Ey (66.10) 
dx = udt + 93 (x1, x2) dw, 


where 1, 2 are r-dimensional vector-valued smooth nonlinear functions with ¢1(0) = @2(0, 0) = 0. Our 
goal is to stabilize the equilibrium x; = 0, x2 = 0 in probability. 
Define the backstepping change of variables 
41 =%1, 
(66.11) 

22 = X2 — a4 (x1); 
where a (x1) is a yet-to-design stabilizing function and 2, is an error variable expressing the fact that x2 is 
not the true control. By Ité’s formula, differentiating z; and zz with respect to time, the complete system 
(Equation 66.10) is expressed in the error coordinates (Equation 66.11): 


dz, = dx, =x. dt+ gi dw =(z2+0;)dt+ gi dw, 


Ooty 107?a) (66.12) 
dz, = dx, — da, = udt 4 gt dw Ox, dx, 5 ax 1 1 dt 
Oa, 1 @oy T 7 OW +f 
= dt —-— dw. 66.13 
(: Oxy = Ox? ie Ge Oxy ae : 


The Lyapunov design for stochastic systems is more difficult than for deterministic systems because of 
1 TOV : a ‘ ; ; 
the term 5 Tr {n aa in Equation 66.2, and it cannot be carried out by using the quadratic Lyapunov 


function V = z? +z as in the deterministic case. We use the following quartic Lyapunov function: 


With it, we obtain that 


3 Oa, 1 O04 3 Oa, Oa} 
LV =Z(z +1) + ~z7 9; 91 +23 | u x To eee (p= Loge men 
1 (Z2 ) 2 1%] #1 2 2 Q) 91 3 °2 ue) Ae P) | | 2 Bx 1 


Since 1 (0) = 0, a will vanish at x, = z; = 0. Thus, by the mean value theorem, «(x )) can be expressed as 


ot (x1) = 0 (Z1) = 21011 (21), (66.15) 
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where @;;(Z1) is a smooth function. Thus, the functions ©) (x1), @2(x1, x2) can be written as follows: 


gi(x1) = 2 9n (a) = aire), (66.16) 
p2(x1, x2) = x1 P21 (1, 2) + X2.22(x1, x2) 
= 21 921 (%1, X2) + (Z2 + 01) 22(%1, 2) 
= 21921 (%1, X2) + 22 22(X1, X2) + 21011 (Z1) G22 (x1, x2) 
> zy Wo (x15 x2) + 22 22(x1, x2) (66.17) 


where ix; Wig, i, k = 1,2, are smooth functions. So, Equation 66.14 can be rewritten as 


LV 3 + s ( & 3 ee af Oa} 1 ay T 
=Zj]2Z24+2z; [a =Z Z| u— ——x— = 
142 1{% 371 nil 2 ae 2 axe 019 
3, T Oa, T ( Oa, 
+ =z —- — —_ — . 66.18 
32 (0! Be G) 2 ax, 1 ( ) 


To handle the first and the fourth terms on the right-hand side of Equation 66.18, we need Young’s 
inequality 
Ty << yxy + yi 
x —||x — ; 
VES ? get y 
where the constant € > 0, the constants p > 1 and q > 1 satisfy (p — 1)(q— 1) = 1, and x, y € R”. Thus, 
for the first term of Equation 66.18, by Young’s inequality, we have 
ee 33,4, lo 
32) < ~eF zt 4 — ah, (66.19) 
4 4c4 
where the constant €; > 0, and for the fourth term, we try to arrange it in the form 


ZW(x1,%2), Y= 3; 


so that it can also be combined into the second and the third terms in Equation 66.18, and be canceled by 
a, and u. Substituting Equations 66.16 and 66.17 into the last term in Equation 66.18 yields 


3 Oa Ou 3 gs T/2 
1 1 
522 (0! = of) (« = wo) = 522 (>: a (>: as : (66.20) 


where 
Ou 
Bai (x1,%2) = Wor — —" U1, 
Oxy 


B22(x1, x2) = Wo. 


The next major step is to separate z, and z2; every term can be handled by either a; or hence u. Hence, 
we rearrange the right term of Equation 66.20 as 


2 T 792 
3 3 3 
522 ( y au ( ) as = 522822822 + 3238 39z1Ba1 + 572 (z1Ba1): (21821), (66.21) 
k=1 k=1 


The first two terms on the right-hand side of Equation 66.21 are already in the desired form. Now we 
concentrate on the last term in Equation 66.21: 
3 Ie 3y5/ 1 ee ee 
2 T 2,2 2 404 2 4 404 4.2 
522 (e121) (z1B21) = 232} 5 > Bay = a (zs + dual] ar y @,2Pa Ta wasn 
j=l j=l j=l j=l 


3r 49 
= 3) 5 Bay + 21 €211- (66.22) 


where B24; is the jth component of the vector Bx. 
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Substituting Equations 66.19 through 66.22 into Equation 66.18, we get 


34 1 3 Joy 10a; 
LV < ~ezit+ 4 (t+ awh +23 | u-—m-— +9) 
45141 2 1 {1 2 wy 2 Dx, 2 2 axe ; Pl 


r 
3r 49 


ee - 3 1 
a5 522822822 + 3238 39z1Ba1 + 2 oa 85); Serra 
1 Si 


O04 10a, 1 3 
= 2 - ax, +1) 2 Oe v1 i4 qa? + 522832820 + 383521621 
1 


3 
+- Ola = Bh +7} (a+; sawinwat 3 —€} Sat Fads). 
Su 


At this point, all the terms can be canceled by u and ay. If we choose them as 


3 4 3r_ ag 
Mel SPO; gr savin yea 
Oa, 10a; rT 1 
u = —C2Z2 + —(z2 + Q)) + Zz sz — 3B55z a 
222 Dx, | ata)+s Ox 9} 91 4a” 2839822 — 363921821 ay Pi 


where constants cj > 0, then the infinitesimal generator of the closed-loop system is negative definite: 


2 
= ) Giz; 
i=1 


which means that the equilibrium z = 0 is globally asymptotically stable in probability. In view of Equation 
66.11, the same is true about x = 0. 
66.4.2 General Recursive Design Procedure 


A systematic backstepping design has been developed for the class of nonlinear systems transformable 
into the strict-feedback form: 


dx; =xj4,dt+) (%) dw, i=1,2,...,.n-1 (66.23) 
dx, = udt + 1 (X,) dw, (66.24) 
where X; = [x),...,x;]", and where (Xj) are vector-valued smooth functions with @;(0) = 0. 


The general design is summarized in Table 66.1. We can show that the Lyapunov function V, = 
; ae Zz, satisfies 


n 
LVn <- ) cizh, 
i=l 


which guarantees that the equilibrium z= 0 is globally asymptotically stable in probability. Thus the 
equilibrium x = 0 is globally asymptotically stable in probability. 


66.5 Output-Feedback Stabilization 


For linear systems, a common solution to the output-feedback problem is a stabilizing state-feedback 
controller employing the state estimates from an exponentially converging observer or filter. However, 
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TABLE 66.1 Backstepping Design for General System (Equations 66.23 and 66.24) 


B=xj-4-1, i=L..unm 


i 
Wie => 1p KHL. i, 
l=k 


i-1 


q i-1 
Paj-1 Oaj—1 


at Sal VT OEY o epi pst = 
tL VT SY ag t Say aie ket iot 
k q=1 j= =k-—] 


34 3 i ee et Ce a a a | Joi} 
re . 3 2 eTe.. 2 yee 2 a2 [aaa 
dies (aie toe a eine > ley ae] 4 


i-] k=i+1 l=1 j=l k=1l=1 “ikl 
WQJ+1 = 1, 
i 
a= 3 Zk Qik 
k=1 


Control Law: 


this approach (“certainty equivalence”) is not applicable to nonlinear systems. The situation is even more 
difficult in the stochastic case where nonlinear exponentially convergent observers are very difficult to 
design. 

Fortunately, classes of problems do exist where output-feedback stabilization can be solved. One such 
class is the output-feedback canonical form (Equation 66.8), for which we present a feedback design in 
this section. 

Consider, for example, the nonlinear stochastic system given by 


dx, = xy dt + fi(y) dt + 9 (y) dw, 
dxy = x3 dt + fr(y) dt + biB(y)udt + 93 (y) dw, 
dx3 = f3(y) dt + boB(y)u dt + 93 (y) dw, 


Yr, 


(66.25) 


where only y is measured, the functions fj(y),i = 1,2, 3 are dependent on the output only and assumed to 
vanish at zero; the polynominal b;s + bo is assumed to be Hurwitz (i.e., bob; > 0), and B(y) 4 0, Vy € R, 
which guarantees that the system has a well-defined relative degree p = 3 — 1 =2. 

The goal is to design the output feedback control law to guarantee that the solution process of the 
closed-loop system is bounded in probability and, when the diffusion terms ¢;(0) = 0, i = 1, 2, 3, the zero 
solution of the closed-loop system is asymptotically stable in the large. 

Since the states x2, x3 are not measured, we first design an observer which would provide exponentially 
convergent estimates of the unmeasured states in the absence of noise. The observer is designed as 


S = ho thily—h) + fil), 
hy = 83 + ko(y — 41) +fav) + bi B(y)u, 
3 = k3(y —X1) + aly) + boB(y)u, 


where kj, ky, and k3 are parameters. 
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Let X = [%1, %2,%3]!, and denote the observer error by xX =x—X = [x1, x2, X3]!. Then we have 


—k; 1 0 
di =|—k, 0 1|xdt+e'(y)dw4 Axdt+o'(y) dw, 
—k; 0 0 


where ply) = [P1, $2, 3]. The parameters k,, ky, and kz are chosen such that the matrix A is Hurwitz. 
Define x = [y, Xo, X3]", and let goi(y) =fi(y); goily, x1) = fily) + kix1, i = 2,3. Then we have the fol- 
lowing dynamics for x: 


d& = Dk dt + Gy, x1) dt + [1, 0, 0]: dt + B(y)Bu dt + H(y) dw, 
where 


1 0 goi(y) 
0 1}, Gy,x1)* | go2ly. x1) |, 
0 0 g03(V,X1) 


B=[0,bi, bol’, H(y) =[ei(y), 0,0)". 


In the design, we employ two similarity transformations. With the first transformation we transform 
vector B into an input vector that has all zero elements except the element b. Taking the first transfor- 
mation as 


1 0 0 
ug Oe rie ia Za 13, 
0 _ 0 1 
by 
we obtain 
dé = Dyé dt + Gy(y, x1) dt + [1, 0, 0] X dt + B(y)B,u dt + Hy(y) dw, 
where 
0 1 0 
Dj\=TDT;'=|0 dy 1 |, 
0 dy —dy 
bo by \* 2 
dyj=—>0, dy=—-(—) =-d 0, 
bh ‘ (2) ws 
801 gly) 
Gi(y, x1) = T1G(y, x1) = b ene 4] gi2%1) |, 
0 


— 5802 + 903 £13 (Vy, X1) 
ak 
B, =7T,B=[0,b,,0]’,  Hi(y) = T:H(y) = HU). 


With the second transformation, we would like to transform the second column of the matrix Dj into the 
first unit vector. Let yn = T2&, where 


Then we obtain that 


dy — D2y dt+ Go(y, x1) dt+ {1, —d, —dy]' X dt + B(y) Bou dt+ p(y) dw, 
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where 
dz, 1 0 
Dy=MD,Ty'=|dy» 0 1 |, dr=dy, dxy=dy, dos =—didy, 
dy3 0 —dyy 
Go(y,41) = G1 = [gi —S11 dui +812, —drogii +. g13l? * [ei(y), 22, X1), 3 Ka) 
By =By, Hy =T.H =[gily),—dugily), -—digily)? 2 thi (y)h3 (y)h3 I". 
Noting that 


we obtain that 
n=y 
N2 = x2 — diy 
m3 = &3 — dix2 + diy. 
Denoting 
c=, 
the following dynamics of lower triangular form are obtained for the n-system: 
dy = day dt + (n2 + X2) dt + gaily) dt +hy(y) dw, 
Anz = (C+ dray + go2(ysx1) + biB(y)u — di1 x2) dt + ho(y) dw, 
d¢ = —d),6 dt + Lyy dt + Lox2 dt + Q(y, x1) dt + Py) dw, 
where 
L, = dy3 = —di,, 
Ly =—d\. =—d},, 
Q(y, 1) = 93(ys 41) = dt ily) — dufi(y) +) + (ks — dike) X1, 
(y) = h3(y) = di, 9 (y). 


We are now ready to design the output feedback stabilizing controller via backstepping. Define the 
error variable 


z=nN2—- ay) = x2 — (duyt+aly)), (66.26) 

where a(y) is a smooth virtual control law to be defined. Then we have 

dk = Ax dt +o" (y) dw, 

d¢ = —dy6 dt + Lyy dt + LoXz dt + Q(y, x1) dt + ®(y) dw, 

dy = M,(y) dt + (z+ a(y) + X2) dt + Vi (y) dw, 

dz = [6+ M2(X1,y, 2) + biB(y)u + Nx2] dt + Wa(y) dw, 
where 

Mi(y) = day t+fily), 
M2(X1, 9,2) = g22(y.%1) + doy — a (y)[n2 + gaily) + diy] — se") 


Wily) =m(y)=9!(), Voy) = N91 (y), 
N(y) = —di — a (y). 
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Since A is Hurwitz, there exists a positive-definite symmetric matrix P satisfying 
A™P+PA=-I. 


Instead of employing a quartic Lyapunov function as in Section 66.4, here we illustrate the possibility 
of employing a Lyapunov function which is “nominally” quadratic, but it incorporates state-dependent 
weights on the quadratic terms. We postulate the following Lyapunov function candidate: 


Tre 1 
Vo = 8%) PX + ——? + y? + E(y)z’, 
2d\1 


where 8 > 0 is a constant to be determined, and smooth function E(y) > 0 is a weighting function to be 
determined. 
Then, we have 


LVz = —S|2|2 + 8Tr( (Pe (y)} —2 + ag lOO) 


1 iy 5 j es 
+ at LoxX2 + Q(y, X1)) + B'(y)Im2 + Mi) +. X2]2* 


+ 2y[z + a+ My(y) + X2] + 2z8(y)[¢ + Mo(x1,y, n2) + bi Bly)u+ Nx] 
PLE(y)z7] 


T T7T 
ap. ze Wie wa . (66.27) 


1 
tury a {rv wry 
The functions Y; can be decomposed by using the mean value theorem as follows: 


Wi (y) = hy (y) = 11 (0) + hi (y)y, 
Wo(y) = W2(0) + Wai (y)y. 


Then we have 


WT = (hs (0) + ha (y)y(hx(0) + ha (yy)? = OKT (0) + [2A (Y) +h, Gy], (66.28) 


1 T P(E (y)z* B"(y)z* 2B'(y)z] Py 
21 [TEI Ge OTE = 31 Le | Fees alka 


ty a"(y)z 28") ] | 3 T 
ae A[s al feo 0 ][ei]} +20. 
1 
2 


(2B"(y) + 48'Q)N(y)) lla)IP2 + EQ) W2¥2. 
(66.29) 
In Equation 66.29, we can cancel out the first term by the choice of a, u. We cannot cancel out the second 
term, but we can bound it in terms of y’, as follows: 
BW. Wz = E[W2(0) + War (y)y] x [W2(0) + War (y)y]" 
$25 ||¥2(0)|)? +28) |?y 


We can design the weighting function E(y), such that E||W2)||7 is sufficiently small by selecting 


K 


E(y) = 
1+ || oi()|/? 


where k is a design parameter. 
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Then we obtain that 


BW WF < 2«||W2(0)||? + 2xy. (66.30) 


In order to deal with “second variation terms” in Equation 66.27, we employ the mean value theorem to 
obtain the representations 


oy) = (0) + yO), P(y) = &(0) + yB(y), 
and 


LpXz + Q(y, 1) = LoX2 + Q(0, 0) + yQ(y) + L3%1 
= [L3, La]x2 + Q(0, 0) + yQ(y). 


Then, we have 


STr{o(y)Pe! (y)} = 8Tr{e(0)P¢! (0)} +8Tr{29(0) PG! (y) + Gy) PG! (y)y}y,_~—: (66.31) 


sq IO" GL) = 5TH") ©0)) + a It20" (BL) +B" (WBy)y}y, 


2d11 diy 
(66.32) 
1 = . 1 4 = 
7, blhy thn + 20,8)] = Ft [ (41 + 20) y+ Hs, Lal + 20,0)] 
4 as 13 +L} 
< (4d}, £0) "[Li + QP y? + £007 + 9 2 
4d;, 
Gea Byiod -l 
+e]! [27 + e267 + (4dte2) 112(0,0)|]?, (66.33) 
yxy = 2yXy < e3%35 +65 'y’, (66.34) 
(S'(y)2? + 25 (y)z) 2 < 0383 + 5! (B'(y)z + 28(y))” 2, (66.35) 
where ¢; > 0,1 = 1,2, 3, are analysis parameters to be chosen appropriately. 


Combining Equation 66.27 with Equation 66.28 through 66.35 we arrive at 
= 2 T 1 tr 2 ~2 
LV2 < —d||X\|? + 8Tr{p(0) Pe (0)} + Fa (0) @(0)} — ¢° + 2e3x3 
11 


T E515 2 —1)=_ 12 2 2 = 2 
+ hy(0)hy +a“ as +e, Wx2ll° + 226° + (4dy,e1) — ||$2(0, 0) |] 
11 
1 oe Se ea 
+ 3g, 2 OW) +8 (y)B(y)yhy + 8Tr{29(0) PP" (y) + GPP" (yyy 
+ (2hy (0); (y) + hi (yh; (y)y)y + (4d? £0)! Li + Q(y) Py? + 0? 
+051 +05! (8 (y)z+28()) 2 + B'(y)Ino + Miz? + 2ylz +a+ Mi(y)] 


1 
+ 2zB(y)[¢ + M2 + bi B(y)u] + 2k|| Y2(0) ||? + 5 (zB"(y) +.42'(y)N(y)) lor (y)II?z + 2ky?. 
(66.36) 


In the above differential inequality, we can select the control law (including virtual control law) and the 
design parameters to cancel out all the unfavorable terms except the constant bias terms. The virtual 
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control a and the actual control u are chosen as 


ay) = (3: =) y—Mi(y) ~ z4-Tel207(0)Y) +B" WB)y} 
ll 


~ er ) 
— bam (OVht () + hah, Oy) — STHl29(OPG" GW) + BUIPHT yy) 
= cos E+ Qi)Py, 


1 = ees 5 ail hy as eta 
BO {; Boz + 6+ M2(%1,y, 2) 4 ao) * 250) (S'(y)z+28(y))° z 
ae ole +MQ)lz+ Fe oy "(y) +48’ (y)N(y)) ior}. 


Choosing the analysis parameters £9, €1, €2, €3, and K, sufficiently small, the analysis parameter 8 sufficiently 
large, and the controller parameter {, sufficiently large, such that 


¢; 48—2e3—¢) + > 0, 
2 2 
+R 
2 
4d, 
t3 = 6; — 2k > 0, 


€] €&9 — €2 > 0, 


and by choosing the control parameter 82 as positive, we obtain that 
LV2 <—cyV2+ 0c, (66.37) 


where 


ead. Zabol > 0, 


; | C1 
cy = min } ———— 
ddmax(P) 


d = 
co. = 8Tr{ p(0)Pp" (0)} + (= +2«N0)) Ilo (O)||? + (442, €1) [42 f: (0) — dur fo(0) +a (0)|12 = 0. 


Thus, the solution process of the closed-loop system is bounded in probability and, moreover, when 
(0) = 0, fi(0) = 0,1 = 1,2, 3, we obtain cp = 0 in Equation 66.37, which guarantees that the origin x = 
X = 0 is stochastically asymptotically stable in the large (and, in fact, also globally asymptotically stable in 
probability). 

For a recursive design procedure for the general output-feedback system in the observer canonical 
form, the reader is referred to [12], as well as to an earlier result in [9] where the output-feedback problem 
without zero dynamics was solved using quartic Lyapunov functions. 


66.6 Adaptive Stabilization in Probability 


In this section, we introduce the idea of stochastic adaptive control in the presence of functional and 
dynamic (nonlinear) uncertainties. Our design is presented in the output-feedback case, with nonlinear 
zero dynamics. 

Consider for example the stochastic nonlinear system of relative degree two, given in the following 
form: 


dxz = fo(xz,y) dt + go (xz) dw, (66.38) 
dx, = (x2 +filxz,y)) dt + gf (xz, y) dw, (66.39) 
dxy = (u+fo(xz,y)) dt + gF (x,y) dw, (66.40) 


y=, 
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where y € R is the measurable output; x, € R™ is the state of unmodeled dynamics, and fj, gj, i = 0, 1,2, 
are uncertain locally Lipschitz functions. 
We assume that the uncertain nonlinear functions satisfy the following linear parameterized upper 
bounds (where the parameters are unknown) and the stochastic unmodeled dynamics are stochastic SISS. 
Al. [fie yl < Foal +¥ G2Uy), Wgile WI <AFWalllxcl) +AFValyl), i= 12, Voz) € 
R™ x R, where 1, Qj2, Wii, and jz are known smooth nonnegative functions, and (0) = 0, 
Wi (0) = 0, and /F, hF, i = 1, 2, are unknown positive constants. 
For the noise of unknown covariance, if it can be formulated as X(t) dw, where w(t) is a standard 
Brownian motion and X(t) is a uniformly bounded deterministic (possibly unknown) function, 
then we can modify A1 as follows: ||g;(xz,y) D()|| < hf Wi (lxzll) + AF Wi2((y)), i = 1,2. 
A2. For Equation 66.38, there exists a C* function V;(x z) and Koo functions a (-), a2(-), a(-), and y(-) 
such as V(xz,y) € R™ x R, 


oy (IlXzl]) < Ve(xz) < &2(Ilxzl), (66.41) 
LV2-(xz) < v(lyl) — allel). (66.42) 

The control objective is to design a smooth adaptive output-feedback controller 
K=O(%y) U=U(Xy), (66.43) 


so that the solution process of the closed-loop systems (Equations 66.38 through 66.43) is bounded 
in probability and moreover, when the drift and diffusion vector fields vanish at the origin, the 
output can be regulated to the origin almost surely. 


66.6.1 Backstepping Design 


We view the overall control system as being composed of the unmodeled dynamics (Equation 66.38) and 
the rest of the controlled plants (Equations 66.39 and 66.40). For the subsystem (Equations 66.39 and 
66.40), we will use the tuning function method to design an adaptive output-feedback controller with 
unknown gain function, where xz is considered as a disturbance input. 

First, we introduce a state-estimator for the subsystems (Equations 66.39 and 66.40): 


1 =% + ai(y — hi), 
Xy =u + ar(y —%1); 


where aj, a2 are constants such that s? + a;s+ a2 is a Hurwitz polynomial. We denote % = [X15 %2]", 
x= *, 7 = max{1,]7,h7,i = 1,2}, and 
—a 1 
2 ~ ; 
—a2 0 


F(xz,y) =[fi (xz), falxe y)I7, 
G(xz, y) =[ 91 (Xz. ¥), B2(xzy)I- 


The complete system is governed by the stochastic differential equations 


dxz = fo(xz,y) dt + gj (xz,y) dw, (66.44) 
- bet, 1 
dx = (48+ pF lee, ») dt+ OX y) dw, (66.45) 
dy = (% + X_ +filxz,y)) dt + gf (xz, y) dw, (66.46) 
dX, = (u+ ag(y — X)) dt, (66.47) 


where Equation 66.44 is the zero dynamics, Equation 66.45 is the observer error system, and Equation 
66.46 and 66.47 are the equations on which the backstepping design will be performed. 
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We start our design by introducing the backstepping transformation 
n=), 2=k,—o1(y,)), 


where $1(y, 1) is a smooth virtual control law to be designed, and lisa parameter estimate. In the 
z-variables, the subsystems (Equations 66.46 and 66.47) is governed by 


dz = (z2 + 61 +l" + fila, y)) dt + gf (xz, y) dw, (66.48) 
, O71 0014 10°o 7 T 
dzy = dx2 9 dy al ldt 5 ay? Zi £1 dt = (u+=Q) 422423424) dt+ 0° dw, (66.49) 
where 
0>1 . 7 1d 7 001 O01 4 
Opec —%1), Q= », 23 =—-—Fx, 
1 By Sy +arly—%i), Q2=—5 By? 8181 — Gy fi, Q3 ras 
0014 0 
QQ4 = 1 l, Oo= De, 
al oy 


Next, we present our design in two steps. 
Step 1. Since the polynomial s* + a1s + a2 is Hurwitz, so is the matrix A. Thus, there exists a positive- 
definite matrix P such that A? P + PA = —I. We start building our Lyapunov function by introducing 


81 eT pay? 1 4 l - 2 
Vi = =—(x' P. — —(l-l)’, 
1 acs Ay ANY 5 A ) 


where 8; > 0 is an analysis parameter, i > 0 is the adaptation gain to be chosen freely by the designer, 
4 4 a 

1 = max 41 ye he | is an unknown constant, and /(t) the estimate of / for which the update law is yet 

to be chosen. 


By Ité formula and Equation 66.48, we obtain that 


28 8 
LV, = 81%! PRX? + ST PH(FT PX) + var Tt (G(QPHR"P + %" PxP)G™} 


. 3 1 ~ A 
+P (+o, +X tf + Syars es (—Di. (66.50) 
0 


We need to deal with the uncertain functions f1, fo, g1, and g2. Toward this end, with the help of the mean 
value theorem we observe that 


Gp (lvl) = (gi2(0) + lylGia(lyl))* < 89500) + 8y* HH (Iyl) < 895 (0) + 8y*Gi2(y), 
VAC) = Chin) + lylbia(lyl))* < 8 (0) + 8y*b5 (ly!) < 8 (0) + By Vi2(y) 


for some smooth nonnegative functions 2, Wi2, $2, and Wj2. In addition, with Young’s inequality, we 
obtain 


3 33,4 log 
VE ay ree 
1 
3.4 1 
3 pk 3 4 ~A 
yx. < —ery 1+ 3%); 
4 4e5 
3 4 Dieg 1 2 
Pfileoy) < | Fn9!+ [ox ) y+ ont Gon) 
No 49 No 


3 3 3 : 
5a gi < ar Aue svi + 12niWizy* + 121 Wt, (0), 
1 
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28) 7p 
<i" Pi(RT PX) < * Stet PIP \xI* + 1 PIP Dot + Supe Yay 


2 
643; ‘i 
+ —FIIPIP D #20), 


i=1 
281 ||P I? +81 11P| 
€ 


Tri Py 14 ; in 
So wi + 4e(281 ||P + 81 ||P||Tr{P}) |x| 


8 
var TH G(2PRR™P + 57 PEP)G} < 
i=1 


28; ||P||2 +8; ||PI|Tr{P} <a « 
re. 11 ll 111 ll PIS tiay! 


e€ : 
i=1 


2 
28; || Pll? +81 || P| Tr{P} 
+8 : > Wi (0). 
i=1 


So, substituting these bounds into Equation 66.50, we obtain 


4 4 3 4 3 
LV <y> @ +—efy+ Sony + ranibay 90) +40)) + (3: ery + sno y+ ') 
0 


i eee oe eee eee ee 1 4 w4 7 
I—Dl — 5 — 81 min(P 
Fas a Pt ga tut oh Tere 1 (P)|x|" + Wi (xz) 
aie ees Te : 
+ é|x|* + Wo+ ee) + 12m Wi2(0) + Go, (66.51) 
0 
where 
Gy) = ool PI Sey (66.52) 
i=1 
2 
‘ 28; |[P||? + 81 ||P Tr{P} 
IG) =8 illP II? |Tr{P} dhe (66.53) 
$1(\xzl) = 8 PPD gi (X21) (66.54) 
e4 aa frie 
‘ 28, |PIZ +8 ae 
Wa llxel) = — wh Xel), (66.55) 
i=1 
is 33) 4 2 2 
c= eF IIPll + 4e (28; ||P]? + 81||/PITr{P}) , (66.56) 
i 32° PZ +8 a 
bo =8— : > 4,0), (66.57) 
i=1 
648 2 
a 1 
$0 = IPI? D2, (66.58) 


i=1 


and ¢, £1, No, and n, are design parameters to be chosen later. 
Weare now ready to select the virtual update law and the virtual control as 


in A 3 4 Es 3 
@(y,1) = —dool 4 ro (Felyt+ morte ‘), (66.59) 
1 
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. 94 9, : 
o1(ys1) = —Biy — vily*)y rae, ye Oey 12H Wi2(y)y 
0 
Ue Pee a ea 
d (Fely+ 5M ¥ + —y} — OY) — Wy), (66.60) 
N1 


where o > 0 is the parameter of the standard “sigma-modification/leakage” and vj (-) isa smooth nonneg- 
ative function to be designed later to dominate various functional and dynamic uncertainties, including 
the effect of the uncertain and unmeasured zero dynamics. It now follows from Equations 66.51, 66.59 
and 66.60 that 


T' 24 i K A 1 1 
LV1 < —Biy* —vily*)y4 + bo —-DUd—w1)-o0 -Di+ 2 ami 4, (Ixzl) 
1 0 


3 - HN 8s ~ 3 wt 
smi (xe) + $1 (|xz|) + reas +.W1([x2]) — 81 amin (P)|X|* + E1x|4 
a 


) es 
t ani2) +1291 W15(0) + Go + Vo, (66.61) 
0 


which completes the first step of the design process. 
Step 2. Consider the Lyapunov function candidate 


Ae 1 
V=Vi(x",y, I) + ra 
By Equation 66.49, and Young’s inequality, we obtain 


Zz 3 3 aT 
LV = LV + (Ut Q1 + 22 + 03 + 4) + 5 2).O2 O2 
0014 
Ry aly 
al 
ao,\2_ .\? BOP. oN 4m 28 
1 1 4 1\4,4 
tl} z +1] z2+—(—)*z 
(2) (2) 2m oy” 


€1 44 1 4 2% 4 % 4 g a 1 oh 
t+rViutpaeiut aeny +4avny + Lnibiy’ + 74%. 
2 4N No 4c} 


1 
<LV, +2 (1 Qa + 7422 
465 


3 2 
+ Smit 2 + 4e, W4,(0) + 12m 4, (0) 
0 


tx x 00h 
Ebi Sie (JD) (1-01) — ol — D1 — 8) amin(P)|XI* + E1X|4 
0 


1 0014 
+ watz (+914 a) -4 1) 
€} 4e5 
1 Po 34 ( (an) 3 4 ( (ab) 3 1 
l 26 3 1] ee etate ] Ay. = of ENA 
FE re "0 (5 agra arved) ieee acre re tec 
a 7 iA. F 
+ bray* + 4erWiay* + 85 + 5 + lay? + 8(\xel) +o, (66.62) 
0 4g} 4e5 
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where 


1 3 
8(|xz|) = ppd Otel) + 3m (heel) + SWi (xe!) + di (lxel) + Hi (leed) 
0 


4 ere 
To= a2) + (4€, + 241) Wt, (0) + Go + Wo, (66.63) 
0 


and €}, €2, and €2 are design parameters to be chosen later. 
We now choose our final parameter update law and control law as 


1= way, 2,1) = wi(y,1) + dozf02, (66.64) 


u = —B222 — Qy 


1 (®b\? 3. $((an\?,\ 3.4 (fan)? ,\’, 3 (ae) 
02 = t 1 t 1 ‘ 66.66 
: aloe) ae ce) a Ce) ala, ee! 


The function v1 (-) is chosen as 


A 0 
22 — lzyb2+ so, (66.65) 


- De, 
vi(y?) = (12) + 4e1) Wi2(y) + = Pi2(y) + vy), (66.67) 
0 
where v(-) is a smooth Koo function which is yet to be chosen in order to deal with the dynamic uncertainty 
coming from the zero dynamics. 
With the choices (Equations 66.64 through 66.67), we obtain 


ae 7 1. ee eee 
LV < —Biy* — Bazi — v(y*)y* — ol — DI 1 rmin(P)IRI + 5 G9 + G3 + ER? + B(lxel) + To. 


£5 Act 
(66.68) 
Using the property of “sigma-modification” that 
spleen (aie? (66.69) 
2 2 
we obtain 
(ome ts 1s ey ere 
LV <—Biy* — Brzz — v(y*)y* — <= 1)? — 8 amin (PYRE? + 55 + 55 + XI 
2 4e5 4ey 
o 
+ a + 8(|xz|) + To. (66.70) 
Recalling Equation 66.55, we obtain 
Oo» % oO 
LV < —Biy* — Bozz — v(y*)y* — SUD? = eo #1? + SP + B(lxel) +o, (66.71) 
where 
34 oa 4 
co= 81 (mint) — Ses IP IP — 4e(2 IPI? + \Pits(e) ) Pie 
i 


i=l 
Choosing the control parameter ¢ sufficiently small, and then choosing the analysis parameter 8) suffi- 
ciently large (for given €2, €;, and €2 of any positive values), we obtain cp > 0. The design parameters no; 
N1> €1> 1, Bz, and Xo are chosen as positive, but otherwise of any value. Hence, we obtain 


LV <—W(%y,z251) — v(y?)y* + (|xel) +P 0 4 SP, (66.72) 


where W(X, y, z251) = 7, Biz + colXI4 + $0 —D?. 
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66.6.2 Dominating the Uncertain and Unmeasured Zero Dynamics 


From Equations 66.42 and 66.72, we know that the subsystems (Equations 66.48, 66.49, 66.64, and 66.65) 
is practically SISS with respect to the state x, of unmodeled dynamics, while the unmodeled dynamics 
(Equation 66.38) are assumed to be SISS with respect to the output y. We need to deal with this stochastic 
feedback connection and its effect on closed-loop stability, by making one final choice for our control 
law, the function v(-) in Equation 66.67. 

Toward this end, we note that there exist known smooth nonnegative functions yz and Wo satisfying 
|e) | < We(|xel) and |lgo(xz)Il < Wo(|xel). If 


gh (s) Wi (s) 


20 )atp2 
< oo, lim sup < oo, lim sup Welsbols) < 00, 
soo4 als) soo+ (5) s>0+ a(s) 


i= 1,2, (66.73) 


where the first two sets of terms appear in Equation 66.63 and a appears in Assumption A2, then we 
define continuous increasing functions &(s) > 0 and ¢(s) > 0 on [0, 00) as E(s) = 4 SUP re(0,s] 8(t)/a(t) for 
s>0 and &(0) = 4lim sup, _,9, 8(t)/a(t), and ¢(s) =2 SUP re(0,s] W2(t) 3 (1) /a(t) for s > 0 and ¢(0) = 


2 lim sup,_,94 W2(t) We (t)/a(t). In this way we obtain 8(s) < &(s)a(s)/4 and W2(s)Wa(s) <¢(s)a(s)/2, 
which we shall use later in selecting the control function v(-). 
If Equation 66.73 holds and if 


OO S — aoe 
/ [E(ay(s))Ver Jol6(r I" gs < 0, (66.74) 
0 


then there exists a nondecreasing positive function p € C'[0, 00) such that for all x € R™, 


(Vz (x))a(|x|) > 20 (Ve(x)) 2 (lal) bo (|x1) + 48(lx1), (66.75) 


where 8(-) is given as in Equation 66.63. The function p, which will be used to modify the Lyapunov 
function, can be constructed as, for example, 


eee aa [&(a,"(s))e7 Jo nat ge 
0 


and 


p(s) = elo nts] 9(0) - i} que fo nae dul, s>0, 
0 


i CO) 
gata’ 2) = paTGy" 
We now consider the Lyapunov function 


where qi(s) = 


= - Vz (xz) és 
V(X XV Za, 1) = / p(s) ds ++ V(x, y, Lz). 
0 
By Ité’s formula we have 


av,\" |’ 
Oxz 80 


F 


= 1, 
LV = pWVz)LVz + 50 (Vz) 


tas “ oO 
— W(X, 9, Z25 1) — v(y*)y* + 8(xz|) +o + ae 


1, x 
< e(Ve)v(Iyl) — allel] + 50 (Vz) W2(Ixel) Wo (Ixel) — W(% y, 2251) — v(y*)y* + 8(|xz]) + Po + A 


1 1, 
< e(n(lyD)v(yD = 5e(Vz)a(lxel) + 50 (Vz) 2 (lxz1) 5 (x21) 


— WO y 2251) — VO?)y* + (xl) +o + SP, (66.76) 
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where n = a2(a7!(2y(-))) € Koo. From Equation 66.75, we have 


1 1, 
zl Ve)a(lael) — B(lxzl) = 50 (Vz) W2(Ixzl) WG (Ixzl)> (66.77) 


thus we get 


— ] ‘A 
LV < e(nily)v(yl) — 7 pV) xe1) — W(ky, 221) — vy”) y* + Po 4 SP. (66.78) 


Assuming that the gain function y(-) in Assumption A2 is locally quartic, that is 


lim sup a < 00, (66.79) 
so0+ S$ 
we choose v(-) as 
2y =] 2 y(s) 
vy") = p(a2(a-(2y(y+1)))) sup =. 
se(0y24+1] 5 
This, together with Equations 66.76 and 66.77, yields 
—_ 1 A 
LV < ~~ p(0)a( lel) — Wy, 2051) +10 + SP. (66.80) 


Define : 
Wi (x, Xz>V> 225 l) = 4 0(0)a(|xz|) + W(x, y, 22, N), 


which is positive-definite and radially unbounded in (x, xz, y, Za; 1). Thus 
= nN oO " o 
LV <—Wi(X, Xz, y, 22,1) + To + 5 < —B(l(% X25 ys 22, D)|) + To + 5h (66.81) 
where f is a class Koo function. 


66.6.3 Boundedness, Stability, and Regulation 


From Equation 66.81, by Test for Stability C, the closed-loop system is practically SISS (and hence the 
solution process is bounded in probability). Moreover, if ~j2(0) = 0, Wi2(0) = 0, i = 1, 2, we obtain’ = 0. 
In this case, we take o = 0 in Equation 66.59; hence from Equation 66.80 we obtain 


LV < —W2(%, xz, y, Z2)s (66.82) 


where W2(X, Xz, y, Z2) = 4 e(0)cu(|xz1) + aes Biz} + co|x|*. Thus, by Test for Stability A, the equilibrium 
(X, z, xo, 1) = (0, 0, 0, 1) is globally stable in probability, and in addition, the output is regulated to the 
origin almost surely, more precisely, P{lims—, o0(|y| + |xz| + |x1| + |x2|) = 0} = 1. 

In this section, we have used a different Lyapunov function from the previous two sections. The basic 
idea of construction comes from the method of changing the supply function of unmodeled dynamics (or 
zero dynamics). For further details and an extension to a general class of systems the reader is referred 


to [14]. 


66.7 Inverse Optimal Stabilization in Probability 


The problem of inverse optimal stabilization in probability for system (Equation 66.6) is said to be solvable 
if there exists a Koo function y2 whose derivative y’, is also a Koo function, a matrix-valued function R2(x) 
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such that Ry (x) = Ry(x)! > 0forallx,a positive definite function /(x), anda feedback control law u = a(x) 
continuous everywhere with «(0) = 0, which guarantees global asymptotic stability in probability of the 
equilibrium x = 0 and minimizes the cost functional 


yu) =| i [I(x) + yo(IRo(x) 2a] act. 


Next, we present a constructive solution to this nonlinear stochastic optimal control problem. 
Consider the control law 


—1/2 
ly2(ILg, VRy /71) 


u= a(x) = —Ry (Lg, V)" -+5 
[Lg VRy (2 


(66.83) 


where V(x) is a Lyapunov function candidate, y2 is a Ko function whose derivative is also a Koo function, 
R>(x) is a matrix-valued function such that R2(x) = RT (x) > 0, and 


ty(r) = i (vy (9) ds 
0 


represents the Legendre-Fenchel transform. Ifthe control law (Equation 66.83) achieves global asymptotic 
stability in probability for system (Equation 66.6) with respect to V(x), then the control law 

io -1/2 
(Ya) VRy "1 
1/2 : 


BR (Lp V) 


_P = B>2 (66.84) 
2 EaVR; 


u* =a*(x)= 


solves the problem of inverse optimal stabilization in probability for system (Equation 66.6) by minimizing 


the cost functional ss 
10) =8| | [a) +62 (Fins) | axl, 
0 


1 Vv re 
))-LyV str {ef al B(B — 2)lya(|Lg VR, 


where 


1/2 1/2 


I(x) = 28 Eee VR, |). 


For the design procedures for control laws in the form (Equation 66.83), and hence of inverse-optimal- 


in-probability control laws in the form (Equation 66.84), the reader is referred to the book by Krsti¢ and 
Deng [9]. 


66.8 Extensions 


In the preceding sections, we introduced stabilization tools for three classes of canonical systems. These 
design ideas can be applied to the following more general systems. 
Systems not in the output-feedback form: 


dxz = fo(xz,y) dt + g4 (xz y) dw, 


dx; = (xi41 + filz,Xi)) dt + gf (xz) dw, i=1....2-1, 
day = (ut fulXer%n)) dt + Bp (Xe»En) dw, 
Y=) 
where nonlinear terms fj, gj,i=1,2,...,n depends on the unmeasurable states x2,...,x; besides the 


measured output y. To design the output-feedback stabilization controller, some extra growth conditions 
on these nonlinear terms are usually needed. 
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Unknown virtual control coefficients: 


dx; = (bixis1 + filXi)) dt +g) &i)dw, i=1,....0-1, 
Axy = (bnut fa(Kn)) dt + g1 (Kn) dw, 


where the constant coefficients b;,i = 1,...,n are unknown, but the signs of bj are assumed to be known. 
Similar to the deterministic case, using the tuning functions design which, in addition to estimating the 
b;’s, also estimates the inverse coefficients be i=1,...,n, we construct adaptive control laws that achieve 


global 


stability in probability and “almost sure” regulation to the origin. 
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67.1 Introduction 


In many applications, such as diffusion and structural vibrations, the physical quantity of interest depends 
on both position and time. Some examples are shown in Figures 67.1 through 67.3. These systems are 
modeled by partial differential equations (PDEs) and the solution evolves on an infinite-dimensional 
Hilbert space. For this reason, these systems are often called infinite-dimensional systems. In contrast, 
the state of a system modeled by an ordinary differential equation evolves on a finite-dimensional system, 
such as R”, and these systems are called finite-dimensional. Since the solution of the PDE reflects the 
distribution in space of a physical quantity such as the temperature of a rod or the deflection of a 
beam, these systems are often also called distributed-parameter systems (DPS). Systems modeled by delay 
differential equations also have a solution that evolves on an infinite-dimensional space. Thus, although 
the physical situations are quite different, the theory and controller design approach is quite similar to 
that of systems modeled by PDEs. However, delay differential equations will not be discussed directly in 
this chapter. 

The purpose of controller design for infinite-dimensional systems is similar to that for finite- 
dimensional systems. Every controlled system must of course be stable. Beyond that, the goals are to 
improve the response in some well-defined manner, such as by solving a linear-quadratic (LQ) optimal 
control problem. Another common goal is design the controller to minimize the system’s response to 
disturbances. 

Classical controller design is based on an input/output description of the system, usually through 
the transfer function. Infinite-dimensional systems have transfer functions. However, unlike the transfer 
functions of finite-dimensional systems, the transfer function is not a rational function. If a closed-form 
expression of the transfer function of an infinite-dimensional system can be obtained, it may be possible 
to design a controller directly. This is known as the direct controller design approach. Generalizations 
of many well-known finite-dimensional stability results such as the small gain theorem and the Nyquist 
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FIGURE 67.1 A flexible beam is the simplest example of transverse vibrations in a structure. It has relevance to 
control of flexible robots and space structures. This photograph shows a beam controlled by means of a motor at one 
end. (Photo courtesy of Prof. M.F. Golnaraghi, Simon Fraser University.) 


FIGURE 67.2 Acoustic noise in a duct. A noise signal is produced by a loudspeaker placed at one end of the duct. In 
this photo, a loudspeaker is mounted midway down the duct where it is used to control the noise signal. The pressure 
at the open end is measured by means of a microphone as shown in the photo. (Photo courtesy of Prof. S. Lipshitz, 
University of Waterloo.) 


FIGURE 67.3 Vibrations in a plate occur due to various disturbances. In this apparatus the vibrations are controlled 
via the piezo-electric patches shown. (Photo courtesy of Prof. M. Demetriou, Worcester Polytechnic University.) 
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stability criterion exist; see [24,25]. Passivity generalizes in a straightforward way to irrational transfer 
functions [9, Chapters V, VI] and just as for finite-dimensional systems, any positive real system can be 
stabilized by the static output feedback u = —xy for any k > 0. For some of these results it is not required 
to know the transfer function in order to ensure stability of the controlled system. It is only required 
to know whether the transfer function lies in the appropriate class. PI-control solutions to tracking 
problems for irrational transfer functions and the internal model principle is covered in [25,26,35]. For 
a high-performance controlled system, a model of the system needs to be used. Hoo-controller design 
has been successfully developed as a method for robust control and disturbance rejection in finite- 
dimensional systems. A theory of robust Hoo-control designs for infinite-dimensional systems using 
transfer functions is described in [11]. More recent results on this approach can be found in [19] and 
references therein. 

The chief drawback of direct controller design is that an explicit representation of the transfer 
function is required. Another drawback of direct controller design is that, in general, the result- 
ing controller is infinite-dimensional and must be approximated by a finite-dimensional system. For 
this reason, direct controller design is sometimes referred to as late lumping since a last step in 
the controller design is to approximate the controller by a finite-dimensional, or lumped parameter, 
system. 

For many practical examples, controller design based on the transfer function is not feasible, since 
a closed-form expression for the transfer function may not be available. Instead, a finite-dimensional 
approximation of the system is first obtained and controller design is based on this finite-dimensional 
approximation. This approach is known as indirect controller design, or early lumping. This is the most 
common method of controller design for systems modeled by PDEs. The hope is that the controller has 
the desired effect on the original system. That this method is not always successful was first documented in 
Balas [1], where the term spillover effect was coined. Spillover refers to the phenomenon that a controller 
which stabilizes a reduced-order model need not necessarily stabilize the original model. Systems with 
infinitely many poles either on or asymptoting to the imaginary axis are notorious candidates for spillover 
effects. However, conditions under which this practical approach to controller design works have been 
obtained and are presented in this chapter. 

In the next section a brief overview of the state-space theory for infinite-dimensional systems is 
given. Some issues associated with approximation of systems for the purpose of controller design 
are discussed in the following section. Results for the most popular methods for multi-input-multi- 
output controller design, LQ controller, and Hoo-controller design are then presented in Sections 67.4 
and 67.5. 


67.2 State-Space Formulation 


Systems modeled by linear ordinary differential equations are generally written as a set of n first-order 
differential equations putting the system into the state-space form 


Z(t) = Az(t) + Bu(t), (67.1) 


where z(t) € R”, A € R”*", B € R"*™, and m is the number of controls. 

We write systems modeled by PDEs ina similar way. The main difference is that the matrices A becomes 
an operator acting, not on R”, but on an infinite-dimensional Hilbert space, Z. Similarly, B maps the 
input space into the Hilbert space Z. More detail on the systems theory described briefly in this section 
can be found in [8]. 

We first need to generalize the idea of a matrix exponential exp(Af). Let £(11, 12) indicates bounded 
linear operators from a Hilbert space | to a Hilbert space 1. 
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Definition 67.1: 


A strongly continuous (Co-) semigroup S(t) on Hilbert space Z is a family S(t) € L(Z, Z) such that 


1. S(O)=I, 
2. S(t)S(s) = S(t+s), 
3. lim;jo S(t)z = z, for all z € Z. 


Definition 67.2: 


The infinitesimal generator A of a Co-semigroup on Z is defined by 
. 1 
Az = lim —(S(t)z — z) 
tho t 
with domain D(A) the set of elements z € Z for which the limit exists. 
The matrix exponential exp(At) is a special case of a semigroup, defined on a finite-dimensional space. 
Its generator is the matrix A. Note that we only have strong convergence of S(t) to the identity J in 


Definition 67.1(3). Uniform convergence implies that the generator is a bounded operator defined on the 
whole space and that the semigroup can be defined as 


(At)! 
ae 


i=0 


just as for a matrix exponential. However, for PDEs the generator A is an unbounded operator and only 
strong convergence of S(t) to J is obtained. 
If A is the generator of a Co-semigroup S(t) on a Hilbert space Z, then for all zy € D(A), 


“ S(t)zo = AS(t)Z = S(t) AZo. 


It follows that the differential equation on Z 


dz(t) 
dt 


= Az(t), z(0)=2Z0 


has the solution 
z(t) = S(t)zo. 


Furthermore, owing to the properties of a semigroup, this solution is unique, and depends continuously 
on the initial data zp. 
Thus, for infinite-dimensional systems, instead of Equation 67.1, we consider systems described by 
dz 
ae Az(t)+Bu(t), z(0)=2Zo (67.2) 
where A with domain D(A) generates a strongly continuous semigroup S(t) on a Hilbert space Z and 
Be LU, Z). We assume also that U/ is finite-dimensional (for instance, R”) as is generally the case in 
practice. 
For some situations, such as control on the boundary of the region, typical models lead to a state- 
space representation where the control operator B is unbounded on the state space. More precisely, it 
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FIGURE 67.4 Heat flow in a rod. The regulation of the temperature profile of a rod is the simplest example of a 
control system modeled by a PDE. 


is a bounded operator into a larger space than the state space. However, this complicates the analysis 
considerably. To simplify the exposition, this chapter considers only bounded B. Appropriate references 
are given for extension to unbounded operators where available. Note, however, that including a model 
for the actuator often changes a simple model with an unbounded actuator to a more complex model 
with bounded control; see, for instance, [17]. 


Example 67.1: Diffusion 


Consider the temperature in a rod of length L with constant thermal conductivity Kg, mass density p 
and specific heat Cp (see Figure 67.4). Applying the principle of conservation of energy to arbitrarily 
small volumes in the bar leads to the following PDE for the temperature z(x, t) at time t at position x 
from the left-hand end (see, e.g., [14, e.g., Section 1.3]) 


0z(x, t) K 02z(x, t) 


G = , xe(0,L), t>0. 67.3 


ot 


In addition to modeling heat flow, this equation also models other types of diffusion, such as 
chemical diffusion and neutron flux. To fully determine the temperature, one needs to specify the 
initial temperature profile z(x,0) as well as the boundary conditions at each end. Assume Dirichlet 
boundary conditions: 


z(0,t)=0, z(L,t)=0. 


and some initial temperature distribution z(0) = Zg, Zp € £L2(0,L). Define Az = 095072 /Ox?. Since we 


cannot take derivatives of all elements of £2(0,L) and we need to consider boundary conditions, 
define 


DIA) = {z € H(0,L); z(0) = 0,z(L) = 0}, 


where #/2(0, L) indicates the Sobolev space of functions with weak second derivatives [37]. We can 
rewrite the problem as 


z(t) =Az(t), z(0,t) = Zo. 


The operator A with domain D(A) generates a strongly continuous semigroup S(t) on Z = £2(0, L). 
The state z, the temperature of the rod, evolves on the infinite-dimensional space £2(0, L). 
Suppose that the temperature on an unit interval [0,1] is controlled using an input flux u(t) 


dz 02z 
—=—.~+Bu(t), 0 (5 
ot aa ay ee 


where B € L(R, Z) describes the distribution of applied energy, with the same Dirichlet boundary 
conditions. Since the input space is one-dimensional, B can be defined by Bu = b(x)u for some 
b(x) € £2(0, 1). This leads to 


z(t) = Az(t) + b(x)u(t). 
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FIGURE 67.5 Simply supported beam with applied force. 


Example 67.2: Simply Supported Beam 


Consider a simply supported Euler-Bernoulli beam as shown in Figure 67.5 and let w(x, t) denote the 
deflection of the beam from its rigid body motion at time t and position x. The control u(t) is a force 
applied at the center with width 8. The analysis of beam vibrations is useful for applications such as 
flexible links in robots; but also in understanding the dynamics of more complex structures. If we 
normalize the variables, we obtain the PDE (see, e.g., [14, Chapter 6]): 


aw tw 
— =bD(x)u(t) t>0, 0<x<1, 

Et ot = boule, t= 
8 
a |x = 0.5| < 5 
b(x) = 5 
0, |x—0.5|> = 
2 

with boundary conditions 
w(0, t) = 0, Wxx (0, t) = 0, w(1, t) = 0, Wyxx (1, t) =0. (67.4) 


This system is second order in time, and analogously to a simple mass-spring system, we define the 
state as z(t) = [w(-, t) W(-, t)]. Let 


Hs(0, 1) = {w € H(0, 1), w(0) = 0, w(1) = 0} 


and define the state space Z = H,(0, 1) x £2(0, 1). A state-space formulation of the above PDE prob- 
lem is 


4 ot) = Az(t) + Bu(t), 


dt 
where 
0 I 
0 
B= ‘ A= Gs F 
| ane. 


with domain 
DA) = {(, W) € Hs(0, 1) x Hs(0, 1); xx € Hs(0, 1)}. 


The operator A with domain D(A) generates a Cg-semigroup on Z. 


Control System Advanced Methods 


Even for finite-dimensional systems, the entire state cannot generally be measured. Measurement of 


the entire state is never possible for systems described by PDEs, and we define 


y(t) = Cz(t) + Eu(t), 


(67.5) 


where CE L(Z,Y) , EE LU, Y), and Y is a Hilbert space. The expression (Equation 67.5) can also 
represent the cost in controller design. Note that as for the control operator, it is assumed that C is a 
bounded operator from the state space Z. The operator E is a feedthrough term that is nonzero in some 


control configurations. 
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The state at any time tf and control u € L2(0, t; U) is given by 


£ 
z(t) = S(t)zo 4 i. S(t — s)Bu(s) ds 
0 


and the output is 
t 
y = CS(t)zZo + cf S(t — t)Bu(t) dt, 
0 
or defining 


g(t) = CS(t)B, 
y(t) = CS(t)Bzy + (g x u)(t) 


where * indicates convolution. 
The Laplace transform G of g yields the transfer function of the system: If z(0) = 0, 


9(s) = G(s)ui(s). 


The transfer function of a system modeled by a system of ordinary differential equations is always rational 

with real coefficients; for example, (2s + 1)/(s” +s + 25). Transfer functions of systems modeled by PDEs 

are nonrational. There are some differences in the systems theory for infinite-dimensional systems [7]. For 

instance, it is possible for the transfer function to have different limits along the real and imaginary axes. 
The definitions of stability for finite-dimensional systems generalize to infinite-dimensions. 


Definition 67.3: 


A system is externally stable or £2-stable if for every input u € £L2(0, 00;U), the output y € L2(0, 00; Y). If 
a system is externally stable, the maximum ratio between the norm of the input and the norm of the output 
is called the Lz-gain. 


Define 
Hy ={G: cr —C|G analyticand sup |G(s)| < oo} 
Res>0 
with norm 
IGlloo = sup |G(s)|. 
Res>0 


Matrices with entries in Hl will be indicated by M(H). The Hoo-norm of matrix functions is 


IGlloo = sup Omax(G(s)). 
Res>0 


The theorem below is stated for systems with finite-dimensional input and output spaces U/ and Y but it 
generalizes to infinite-dimensional U/ and Y. 


Theorem 67.1: 


A linear system is externally stable if and only if its transfer function matrix G € M(Hoo). In this case, 
|Glloo is the L2-gain of the system and we say that G is a stable transfer function. 


As for finite-dimensional systems, we need additional conditions to ensure that internal stability and 
external stability are equivalent. 
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Definition 67.4: 


The semigroup S(t) is exponentially stable if there is M > 1,a > 0 such that ||S(t)|| < Me“ for all t > 0. 


Definition 67.5: 


The system (A, B, C) is internally stable if A generates an exponentially stable semigroup S(t). 


Definition 67.6: 


The pair (A, B) is stabilizable if there exists K € L(U, Z) such that A — BK generates an exponentially stable 
semigroup. 


Definition 67.7: 


The pair (C, A) is detectable if there exists F € L(Y, Z) such that A — FC generates an exponentially stable 
semigroup. 


Theorem 67.2: [18, Theorem 26, Corollary 27] 
A stabilizable and detectable system is internally stable if and only if it is externally stable. 


Now, let G be the transfer function of a given plant and let H be the transfer function of a controller, 
of compatible dimensions, arranged in the standard feedback configuration shown in Figure 67.6. This 
framework is general enough to include most common control problems. For instance, in tracking, r is 
the reference signal to be tracked by the plant output y;. Since r can also be regarded as modeling sensor 
noise and d as modeling actuator noise, it is reasonable to regard the control system in Figure 67.6 as 
externally stable if the four maps from r, d to e1, e2 are in M(H.). (Stability could also be defined in terms 
of the transfer matrix from (r, d) to (y1, yz): both notions of stability are equivalent.) Let (A, B, C, E) be a 
state-space realization for G and similarly let (A;, B,, C;, E-) be a state-space realization for H. If I+ EE, 
is invertible, then the 2 x 2 transfer matrix A(G, H) which maps the pair (r, d) into the pair (e1, e2) is 


r e u eg y 


FIGURE 67.6 Standard feedback diagram. 
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given by 
-1 = -1 
A(G.H) = ie GH) G(I + HG) 


H(I+GH)7! ([+HG)7! 


Definition 67.8: 


The feedback system (Figure 67.6), or alternatively the pair (G, H), is said to be externally stable if I + EE, 
is invertible, and each of the four elements in A(G, H) belongs to M(H). 


Typically the plant feedthrough E is zero and so the invertibility of I+ EE, is trivially satisfied. The 
above definition of external stability is sufficient to ensure that all maps from uncontrolled inputs to 
outputs are bounded. Furthermore, under the additional assumptions of stabilizability and detectability, 
external stability and internal stability are equivalent. 


Theorem 67.3: [18, Theorem 35] 


Assume that (A,B,C,E) is a jointly stabilizable/detectable control system and that a controller 
(A¢; Bc, Cc, Ec) is also jointly stabilizable/detectable.The closed-loop system is externally stable if and only 
if it is internally stable. 


This equivalence between internal stability and external stability justifies the use of controller design 
techniques based on system input/output behavior for infinite-dimensional systems. 


67.3 Issues in Controller Design 


For most practical examples, a closed-form solution of the PDE or of the transfer function is not available 
and an approximation needs to be used. This approximation is generally calculated using one of the many 
standard methods, such as finite elements, developed for simulation of PDE models. The resulting system 
of ordinary differential equations is used in controller design. The advantage to this approach is that the 
wide body of synthesis methods available for finite-dimensional systems can be used. 

The usual assumption made on an approximation scheme used for simulation are as follows. Suppose 
the approximation lies in some finite-dimensional subspace Z,, of the state space Z, with an orthogonal 
projection P, : Z —> Z, where for each z € Z, limy—o0 || Pnz — z|| = 0. The space Z, is equipped with 
the norm inherited from Z. Define B, = P,B,C, = C|z, (the restriction of C, to Z,) and define Ay, € 
L(Zn, Zn) using some method. This leads to a sequence of finite-dimensional approximations 

dz 
a Anzlt) + Bult), (0) = Przo, 
y(t) = Cnz(t). 
Let S, indicate the semigroup (a matrix exponential) generated by A,. The following assumption is 
standard. 


(A1) For each z € Z, and all intervals of time [f), to], 


lim sup ||Sn(#)Pnz— S(t)z|| = 0. 


MO telt t2] 


Assumption (A1) is required for convergence of initial conditions. Assumption (A1) is often satisfied 
by ensuring that the conditions of the Trotter-Kato Theorem hold; see, for instance, [34, Section 3.4]. 
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Assumption (A1) implies that P,z — z for all z € Z. The strong convergence P,z — z for all z € Z and 
the definitions of B, = P,B and C, = C|z, imply that for all ue U,z € Z, ||B,u — Bul| > 0, ||CyPyz— 
Cz|| > 0. 

The following result on open-loop convergence is straightforward. 


Theorem 67.4: 


Suppose that the approximating systems (Ay, Bn, Cy) satisfy assumption (A1). Then for each initial condition 
z € Z,and time T > 0 the uncontrolled approximating state z(t) converges uniformly on bounded intervals 
of time [0, T] to the exact state. Also, for each u € £L2(0, T;U), vn > y in the norm on L2(0, T; Y). 


Example 67.3: Indirect Controller Design for Simply Supported Beam. (Eg. 67.2 cont.) 


Consider a simply supported Euler-Bernoulli beam (Figure 67.5). As shown in Example 67.2, a state- 
space formulation with state space Z = H,(0, 1) x £2(0, 1) is 


d 
zr z(t) = Az(t) + Bu(t), 


where 
A ° 4 pel 
=> d+ ; = , 
a b(.) 
with domain 
D(A) = {(, 1h) € Hs(0, 1) x Hs(0, 1); 6” € Hs(0, 1)} 
and 
! |x-O05|< } 
bx) = 48 2 
0, |x —0 5| eS 
2 


Let ¢j(x) indicate the eigenfunctions of 04w/@x* with simply supported boundary conditions 
(Equation 67.4). Defining Xn to be the span of $;,i=1...n, we choose Zp = Xn x Xp and define 
Pr to be the projection onto Zp. Let (-,-) indicate the inner product on Z (and on Zp). Define the 
approximating system (An, By) by the Galerkin approximation 


(z(t), oj) = (Az(t), 6;) + (6, oj)u(t), i =1...0n. 
This leads to the standard modal approximation 
Z(t) =Anz(t)+ Bnu(t), Zn(0) = Pnz(0). (67.6) 


This approximation scheme satisfies assumption (A1). 
Consider LQ controller design [30]: find a control u(t) so that the cost functional 


CO 
Su,2) = [ (zit), z(0)) + |u(t)|2dt 
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is minimized where z(t) is determined by Equation 67.6. The resulting optimal controller is u(t) = 
—Knz(t), where Ky = BA T1nz(t), and Tp solves the algebraic Riccati equation 


A; Un + TInAn 24 TnBnB; On + | => 0 


where M* indicates the adjoint operator of M. For a matrix M, M* is the complex conjugate transpose 
of M. Suppose we use the first three modes (or eigenfunctions) to design the controller. As expected, 
the controller stabilizes the model used in design. However, Figure 67.7b shows that if even one 
additional mode is added to the system model, the controller no longer stabilizes the system. This 
phenomenon is often called spillover [1]. Figure 67.8 shows the sequence of controllers obtained for 
increasing model order. The controller sequence is not converging to some controller appropriate for 


(a) Center position of controlled beam, three modes 
0.15 T r r r r r T T T 


0.1 + 4 


Amplitude 


) 0.5 1 LS 2 25 3 3.5 4 4.5 5 
Time (s) 


(b) 15 Center position of controlled beam, four modes 


Amplitude 


-15/ 4 


-29 La 
0 05 1 15 2 25 3 35 4 45 5 


Time (s) 


FIGURE 67.7. An LQ feedback controller K was designed for the beam in Example 67.3 using the first three modes 
(eigenfunctions). Simulation of the controlled system with (a) three modes and (b) four modes is shown. The initial 
condition in both cases is the first eigenfunction. Figure (b) illustrates that the addition of only one additional mode 
to the model leads to instability in the controlled system. 
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FIGURE 67.8 LQ optimal feedback for modal approximations of the simply supported beam in Example 67.3. Since 
the input space U = R, the feedback operator K;, is a bounded linear functional and hence can be uniquely identified 
with a function, called the gain. The upper figure shows the feedback gain for the position of beam; the lower figure 
shows the velocity gains. Neither sequence is converging as the approximation order increases. 


the original infinite-dimensional system. The increase in ||Kn|| as approximation order increases sug- 
gests that the original system is not stabilizable. This is the case here. Although the approximations 
are stabilizable, the original model is not stabilizable [13]. 


As shown by the above example, and also by an example in [29], requirements additional to those 
sufficient for simulation are required when an approximation is used for controller design. The issues are 
illustrated in Figure 67.9. Possible problems that may occur are the following: 


¢ The controlled system may not perform as predicted. 

e The sequence of controllers for the approximating systems may not converge. 

¢ The original control system may not be stabilizable, even if the approximations are stabilizable. 

¢ The performance of the controlled infinite-dimensional system may not be close to that predicted 
by simulations with approximations (and may be unstable). 


Although (A1) guarantees open-loop convergence, additional conditions are required in order to obtain 
closed loop convergence. 

The gap topology, first introduced in [40], is useful in establishing conditions under which an approx- 
imation can be used in controller design. Consider a stable system G; that is, G € M(H). A sequence 
Gn converges to G in the gap topology if and only if limy— oo [|G — Glloo = 0. The extension to unsta- 
ble systems uses coprime factorizations. Let G be the transfer function of a system, with right coprime 
factorization G = ND7!: 

XN+YD=I1, X,N,Y,De M(H). 


(If G € Ho, then we can choose N=G,D=1LX=LY 0.) A sequence G, converges to G in the gap 
topology if and only if for some right coprime factorization (Nas Dn) of Gas Nn converges to N and D,, 
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(b) 


G, G 
(a) 
U—| GG, | Yn u—>y G —> y 
A A 
————_> 
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Open-loop convergence Closed-loop convergence 
() (d) 
> G > G G, hig G 
ia, H H, <—Oc H \—O<« 
> Sawa 
Convergence of controllers Convergence of controllers and plants 


FIGURE 67.9 Typical approximation criteria ensure that the open loops converge. However, in controller design, 
the controller is generally implemented as a feedback controller and control of the resulting closed-loop system 
is needed. Furthermore, a sequence of controllers is produced by applying a controller synthesis technique to the 
approximations. This sequence of controllers should converge so that closed-loop performance with the original 
plant is similar to that predicted by simulations with the approximations. 


converges to D in the Ho-norm. For details on the gap (or graph) topology, see [39,42]. The importance 
of the gap topology in controller design is stated in the following result. 


Theorem 67.5: [39] 


Let Gy € M(Hoo) be a sequence of system transfer functions. 


1. Suppose G, converges to G in the gap topology. Then if H stabilizes G, there is an N such that H 
stabilizes Gy, for alln > N and the closed-loop transfer matrix A(Gn,H) converges to A(G,H) in 
the Ho9-norm. 

2. Conversely, suppose that there exists an H that stabilizes G, for all n> N and so that A(Gn, H) 
converges to A(G, H) in the Hoo-norm. Then Gy, converges to G in the gap topology. 


Thus, failure of a sequence of approximations to converge in the gap topology implies that for each 
possible controller H at least one of the following conditions holds: 


¢ H does not stabilize G, for all n sufficiently large. 
¢ The closed-loop response A(G,, H) does not converge to A(Gy, H). 


On the other hand, ifa sequence of approximations does converge in the gap topology, then the closed-loop 
performance A(G,, H) converges and moreover, every H that stabilizes G also stabilizes G,, for sufficiently 
large approximation order. 

The following condition, with assumption (A1) provides a sufficient condition for convergence of 
approximations in the gap topology. It was first formulated in [3] in the context of approximation of LQ 
regulators for parabolic PDEs. 
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Definition 67.9: 


The control systems (An, Bn) are uniformly stabilizable if there exists a sequence of feedback operators 
{Ky} with ||Kn|| <M for some constant M, such that Ay, — ByKy generate Sxn(t) where ||Sxn(t)|| < 
Mpe~%2*, Mo >1,a.>0. 


Since U is assumed finite-dimensional, B is a compact operator. Thus, (A1) and uniform stabilizability 
imply stabilizability of (A, B) [15, Theorem 2.3] and in fact the existence of a uniformly stabilizing sequence 
K, satisfying Definition 67.9 with limy_,o9 Ky Pnz = Kz for all z € Z. The beam in Example 67.3 is not 
stabilizable and therefore there is no sequence of uniformly stabilizable approximations. 


Theorem 67.6: [29, Theorem. 4.2] 


Consider a stabilizable and detectable control system (A, B,C,E), and a sequence of approximations 
(An; Bn, Cn, E) that satisfy (Al). If the approximating systems are uniformly stabilizable then they con- 
verge to the exact system in the gap topology. 


Example 67.4: Diffusion (Eg. 67.1 cont.) 


dz Oz 
Ot Ax2 
z(0,t) = 0, z(1,t)=0, 


+ b(x)u(t), O<x <1, 


1 
vit) = f c(x)z(x, t) dx 
0 


for some b,c € £L2(0, 1). This can be written as 
z(t) = Az(t) + Bu(t), 
y(t) = C2(0), 


where A and B are as defined in Example 67.1 and 
1 
cz= [ c(x)z(x) dx. 
¢) 


The operator A generates an exponentially stable semigroup S(t) with ||S(t)|| < et on the state- 
space £ (0, L) [8] and so the system is trivially stabilizable and detectable. The eigenfunctions ;(x) = 
/2 sin(1ix),i = 1,2,..., of A form an orthonormal basis for £>(0, L). Defining Zp = spanj—1._nj(x), 
and letting Pn be the projection onto Zn, define An by the Galerkin approximation 


(Andj, 0)) = (Adj, o)) f=1...n, f=1...n. 


It is straightforward to show that this set of approximations satisfies assumption (A1) and that 


the semigroup generated by Ap has bound Sp(t) < emt, Hence the approximations are uniformly 
stabilizable (using Kn =0). Thus, the sequence of approximating systems converges in the gap 
topology and will yield reliable results when used in controller design. 


It is easy to show that if the original problem is exponentially stable, and the eigenfunctions of A form 
an orthonormal basis for Z, then any approximation formed using the eigenfunctions as a basis, as in the 
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previous example, will both satisfy assumption (A1) and be trivially uniformly stabilizable and detectable. 
However, in practice, other approximation methods, such as finite elements, are often used. 

Many generators A for PDE models can be described as follows. Let V be a Hilbert space that is dense 
in Z. The notation (-,-) indicates the inner product on Z, and (-,-)y indicates the inner product on V. 
The norm on Z is indicated by || - || while the norm on V will be indicated by ||-|| y. Let the bilinear form 
a:V x Vt>C be such that for some c; > 0 


la(os W)| < cilldlivilllv (67.7) 


for all b, y € V. An operator A can be defined through this form by 


(Ad, b) =—a(d,), Ve V 


with D(A) = {o € V| a(,-) € Z}. Assume that in addition to Equation 67.7, a(-.,-) satisfies Garding’s 
inequality: there exists k > 0, such that for alld € V 


Re a(, ) +k(o, ) > cll oll. (67.8) 


For example, in the diffusion example above, V = H4(0, 1) and 


1 
ao) = 6! ()W'(e) de. 


The inequalities (Equations 67.7 and 67.8) guarantee that A generates a Co- semigroup with bound 
|| T(t)|| < e [37, Section 4.6]. This framework includes many problems of practical interest such as 
diffusion and beam vibrations with Kelvin-Voigt damping [29]. 
Defining a sequence of finite-dimensional subspaces Z, C V, the approximating generator A, is 
defined by 
(AnZn> Vn) = —A(Zns Vn), WZns Vn € Zn. (67.9) 


This type of approximation is generally referred to as a Galerkin approximation and includes 
finite-elements as well as many other popular approximation methods. For such problems, the following 
result, which generalizes [3, Lemma 3.3] is useful. It applies to a number of common applications, such as 
the usual linear spline finite-elements for approximating the heat equation and other diffusion problems. 
Finite-element cubic spline approximations to damped beam vibrations are also included. 


Theorem 67.7: [29, Theorem 5.2,5.3] 


Let Hy C V be a sequence of finite-dimensional subspaces such that for all z € V there exists a sequence 
Zn © Zy with 
lim ||Z, —zlly =0. (67.10) 
noo 


If the operator A satisfies the inequalities (Equations 67.7 and 67.8) then 


1. Assumption (A1) is satisfied with ||S,(t)|| < elt; 
2. IfK € L(Z, UV) is such that A — BK generates an exponentially stable semigroup then the semigroups 
Snk(t) generated by An — ByKPn are uniformly exponentially stable. In other words, there exists 
M > 1,a> 0 such that 
Sux (t)|| < Me“ Wn >QN. (67.11) 


and the approximations (Ay, By) are thus uniformly stabilizable. 
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Example 67.5: Damped String 


The wave equation 
02wix, t) 2 02wix,t) 
ate Ox2 
describes the deflection z(x, t) of a vibrating string of unit length as well as many other situations 
such as acoustic plane waves, lateral vibrations in beams, and electrical transmission lines (see, 
e.g., [38, Chapter 1]). Suppose the ends are fixed: 


, O<x<1, t>0, 


w(0,t)=0, w(1,t)=0. 


Including control and observation, as well as the effect of some light damping [36] leads to the model 


62w(x, t) Ow(.,t) 02w 
D2 + €( at , B(-))b(x) = aul + b(x)u(t), O<x<1, 
a, Ow(x, t) 
y(t) = fo Ox) Ai dx 


where € > 0, and b € £2(0, 1) describes both the control and observation action, which is a type of 
distributed colocation. The state space is Z = HO, 1) x £2(0, 1) and the state-space equations are 


Z Z 
< dz} =A} dz|+ Ea u(t), 
dt dt 
Z 
y(t)=C} dz 
dt 
where 
- 0 | 
A =| @2 
H = —€(v, b) B(x) 
DIA) = {(w,v) € HG (0, 1) x A300, 1}, 
fe "| =[0 (b),v)] 
Suppose that 
1 O0<x< Me 
bx) = 1 7 
0, 5 <x<l 


The eigenvalues Xn of A have all negative real parts, but asymptote to the imaginary axis so that 
supp Re(An) = 0. The results in [18] (see also [8, Section 5.2]) imply that the system is not exponentially 
stabilizable. Thus, no sequence of approximations is uniformly stabilizable. 

However, it is possible to construct a sequence of finite-dimensional approximations that converge 
in the gap topology. The transfer function is 


5sinh(s) + 2cosh(5) — 3cosh(5) + 


1 
$8) = S54 H)sinh(s) + elacosh(S) — 3cosh?(3) +1) 


The function G € Hy; so the system is L>-stable, and furthermore, 


lim G(s) = 0. 
|s| > 0o,Ress>0 
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Thus, we can find a sequence of rational functions Gry so that 
lim ||Gn(s) — G(s)|loo = 0. 
n->oo 


The state-space realizations corresponding to {Gn} are finite-dimensional and thus there are finite- 
dimensional approximations that converge in the gap topology. 


The above example illustrates that uniform stabilizability is a sufficient, not necessary, condition for 
convergence of approximations in the gap topology. 

Once an approximation scheme that converges in the gap topology is found (typically, by finding 
one that satisfies (A1) and is uniformly stabilizable), the next step is controller design. The sequence of 
controllers designed using the approximations should converge to a controller for the original infinite- 
dimensional system that yields the required performance, as well as stability (see Figure 67.9). 

A common procedure for controller design is to first design a state feedback controller: u(t) = —Kz(t). 
Then, since the full state is not available, an estimator is designed to obtain an estimate of the state using 
knowledge of the output y and the input u. The controller is formed by using the state estimate as input 
to a state feedback controller. Controller design of this type for infinite-dimensional systems is described 
in [8, Section 5.3]. 

However, typically both the state feedback and the estimator are designed using a finite-dimensional 
approximation (Ay, By, Cy, E). Suppose F, € L(Y, Z) is found so that all the eigenvalues of A, — F,Cy 
have negative real parts, and similarly K, € £(Z,U) is such that all the eigenvalues of An — By, Ky, have 
negative real parts. The resulting finite-dimensional controller is 


Z(t) = AnZc(t) + Byu(t) oie F,(y(t) _ Cnzc(t)) 
yc(t) = —Knzc(t) 


This framework does not include the effect of disturbances to the controlled system, shown as r and 
v in Figure 67.6. To include these effects, and put the system into the standard framework, define an 


augmented system output 
- C E 
WO = ba = B 2(t)+ H u(t) 


Letting e(t) = r — y indicate the controller input, the controller equations are then 


Z(t) = (An — FyCn)Zc(t) — [Fn By] e(t), 


ye(t) = —Knz-(t) (67.12) 


and the plant input is y, + v. 

It is well known that such a controller stabilizes the approximation (Ay, By, Cy, E) (see, e.g., [30]). 
However, it must also stabilize the original system (A, B, C, E). For this to happen, the controller sequence 
must converge in some sense. For controller convergence, an assumption in addition to (A1) and uniform 
stabilizability is required. 


Definition 67.10: 


The observation systems (An, Cy) are uniformly detectable if there exists a sequence of operators {F,} with 
|Full < M3 for some constant M3 such that Ay — F,Cy generates Spn(t), where ||Spn(t)|| < Mae", Mg > 1, 
and a4 > 0. 
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The approximating systems (A, C) are uniformly detectable if and only if (A%, C#) is uniformly sta- 
bilizable and thus uniform detectability can be established using conditions for uniform stabilizability. 
In particular, if A is defined through a bilinear form satisfying Equations 67.7 and 67.8 and condition 
(Equation 67.10) in Theorem 67.7 is satisfied, then detectability of (A, C) implies uniform detectabil- 
ity of (An, Cy). 


Theorem 67.8: 


Assume that (Al) holds, that the operators Kn, F, used to define the sequence of controllers (Equa- 
tion 67.12) satisfy Definitions 67.9 and 67.10 respectively and there exists K € LU, Z), F € L(Y, Z) such 
that limy—oo KyPnz = Kz for all z € Z, limp oo Fny = Fy for all y € Y. Indicating the controller transfer 
function by H,, the controllers converge in the gap topology and so for sufficiently large n, the output feedback 
controllers (Equation 67.12) stabilize the infinite-dimensional system (Equations 67.2, 67.5). Furthermore, 
the closed loop systems A(Gn, Hn) converge uniformly to the closed system A(G, H) where H indicates the 
transfer function of the infinite-dimensional controller 


Z(t) = (A—FC)z-(t)—[F Ble(t), 


(67.13) 

yc(t) = —Kz,(t). 
Proof. The assumptions on the controller imply that, as for the plant (Theorem 67.6) the controllers 
converge in the gap topology. See [28] for details. This implies that closed loops A(G, H,,) converge to 
A(G, H) and that the controllers stabilize the original system for large enough n. Since the assumptions 
also imply that the approximating systems G, converge in the gap topology to G, the closed-loop systems 
A(G,, Hy) converge uniformly to the closed system A(G, H). | 


Controller design is explored further in the next two sections for the synthesis methods most commonly 
used for multi-input-multi-output systems: LQ control and Ho-control. 


67.4 LQ Regulators 


Consider the LQ controller design objective of finding a control u(t) so that the cost functional 


J(u, zo) = ‘; (Cy z(t), Cyz(t)) + (u(t), Ru(t)) dt (67.14) 
0 


is minimized where R € L(U,U) is a symmetric positive-definite operator weighting the control, C; € 
L(Z, Y) (with Hilbert space ) weights the state, and z(t) is determined by Equation 67.2. The theoretical 
solution to this problem is similar in structure to that for finite-dimensional systems [8,12,22,23]. 


Definition 67.11: 


The system (Equation 67.2) with cost (Equation 67.14) is optimizable if for every zo € Z there exists 
u € L2(0, 00;U/) such that the cost is finite. 


Theorem 67.9: [8, Theorem 6.2.4, 6.2.7] 


If Equation 67.2 with cost (Equation 67.14) is optimizable and (A, C,) is detectable, then the cost function 
(Equation 67.14) has a minimum for every zo € Z. Furthermore, there exists a self-adjoint nonnegative 
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operator TI € £(H, H) such that 


min J(u, Zo) = (20, Iz). 
ueL2(0,c0;U) 


The operator I] is the unique nonnegative solution to the operator equation 
(Az, T1zp) + (T1z1,.Az2) + (C121, C122) — (B*T1z,,R-'B*T1z1) =0 21,22 € D(A). (67.15) 


Defining K = R~'B*1, the corresponding optimal control is u = —Kz(t) and A — BK generates an expo- 
nentially stable semigroup. 


It is straightforward to show that the assumption of optimizability in Theorem 67.9 is equivalent to 
stabilizability. 
The Riccati operator equation 67.15 is equivalent to 


(A*I + TIA — MBR™'B*I1+ CfC\)z=0, Vz D(A). 


In practice, the operator Equation 67.15 cannot be solved and the control is calculated using an approxi- 
mation. The cost functional becomes 


J(u; Zo) = i (Cinz(t), Cinz(t)) + (u(t), Ru(t)) dt (67.16) 
0 


where z(f) is the state of the approximating system 
Z(t) =Anz(t)+Bnu(t), 2(0) = PrZo; 


on Z, and Cy, = Cj|z,. If (An, By) is stabilizable and (Ay, Ci,) is detectable, then the cost functional has 
the minimum cost (P20, MnPnzo) where I, is the unique nonnegative solution to the algebraic Riccati 
equation 


A* Ty + WnAn — WnBaR BY + Ct,Cin = 0 (67.17) 


on the finite-dimensional space Z,. The feedback control K, = R-1B*Ty is used to control the original 
system (Equation 67.2). 

The sequence of controllers K,, along with the associated performance must converge in some sense 
in order for this approach to be valid. Assumption (A1), along with uniform stabilizability, guarantees 
convergence, of the approximating systems. However, in order to obtain controller convergence, a set of 
assumptions involving the dual system (A*, B*, Cy) is required. 


(A1*) 1. For each z € Z, and all intervals of time [f1, f2] 


sup ||S*(t)Pnz — S*(t)z|| > 0; 


te[t tz] 


2. ForallueU,ye Y,||Cj,y — Cfyll > 0 and ||BP,z — B*z|| > 0. 


Theorem 67.10: [3, Theorem 6.9], [15, Theorem 2.1, Corollary 2.2] 


If assumptions (Al), (Al*) are satisfied, (An, By) is uniformly stabilizable and (Ayn, Cin) is uniformly 
detectable, then for each n, the finite-dimensional ARE (Equation 67.17) has a unique nonnegative solution 
II, with sup ||TIn|| < 00. There exists constants M, > 1,0, > 0, independent of n, such that the semigroup 
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Snk(t) generated by Ay — ByKy satisfy 
Sux (t)Il < MyeW"". 


For sufficiently large n, the semigroups Sxn(t) generated by A— BK, are uniformly exponentially stable; 
that is there exists Mz > 1,2 > 0, independent of n, such that 


II Skn(t)|| < Mae". 


Furthermore, letting 11 indicate the solution to the infinite-dimensional Riccati equation 67.15, for all 
ZEZ, 
lim ||,P,z —Tz|| =0 
n> CO 


and 
lim ||KnPnz— Kz|| =0, 
n—>co 


and the cost with feedback K,z(t) converges to the optimal cost: 
J(—Knz(t), 20) > (T1zo; Zo). 


The assumption (A1*) implies open-loop convergence of the dual systems (A%, Cf,,, B®). It is required 
since the optimal control Kz relates to an optimization problem involving the dual system. Note that 
(A, C;) is uniformly detectable if and only if (A*, C{) is uniformly stabilizable, and so (A1*) along with 
uniform detectability can be regarded as dual assumptions to (A1) and uniform stabilizability. Since the 
operators B and C) are bounded, (A1*2) holds if both the input and output spaces are finite-dimensional. 
However, the satisfaction of (A1*1), strong convergence of the adjoint semigroups, is not automatic. 
A counterexample may be found in [5] where the assumptions except (A1*1) are satisfied and the 
conclusions of the above theorem do not hold. The conclusions of the above theorem, that is, uniform 
boundedness of II, and the uniform exponential stability of S,x(t), imply uniform stabilizability of 
(An; Bn). Although Example 67.5 illustrated that uniform stabilizability is not necessary for convergence 
of the approximating systems, it is necessary to obtain an LQ controller sequence that provides uniform 
exponential stability. The above result has been extended to unbounded control operators B for parabolic 
PDEs, such as diffusion problems [2,22]. 


Example 67.6: Damped Beam 


As in Examples 67.2 and 67.3, consider a simply supported Euler—Bernoulli beam but now include 
viscous damping with parameter cg = 0.1. We obtain the PDE 


O2w Ow O4w 


D2 +Cq at + x4 =b-(x)u(t), t>0,0<x<1, 


with the same boundary conditions as before. However, we now consider an arbitrary location r for 
the control operator b so that 


1, Ix—rl< 


NI1o N| oo 


0, Ix-r= 


Recall that the state space is Z = H,(0, 1)(0, 1) x £2(0, 1) with state z(t) = (w(., t), ow., t)). An 
obvious choice of weight for the state is C; =/. Since there is only one control, choose control 
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FIGURE 67.10 Optimal actuator location and performance for approximations of the viscously damped beam with 
weights C; = I, R = 1. No convergence of the optimal location or performance is seen as the approximation order is 
increased. 


weight R = 1. We wish to choose the actuator location in order to minimize the response to the 
worst choice of initial condition. In other words, choose r in order to minimize 


max min J" (u,Z9) = || M( I]. 
29 €Zllzq|=1 UEL2(0,00;U) 


The performance for a particular r is w(r) = || I1(r)|| and the optimal performance 


i= inf |M()]. 
ph satay (r)|| 


This optimal actuator location problem is well-posed and a optimal location fF exists [31, Theorem 2.6]. 
Let 4;(x) indicate the eigenfunctions of 04w/dx* with simply supported boundary conditions. 
Defining Xp to be the span of $;,i=1...n, we choose Zp = Xp x Xp. This approximation scheme 
satisfies all the assumptions of Theorem 67.10 and so the sequence of solutions [Ip to the corre- 
sponding finite-dimensional AREs converge strongly to the exact solution I. 
However, as shown in Figure 67.10, this does not imply convergence of the optimal actuator 
locations, or of the corresponding actuator locations. 


The problem is that strong convergence of the Riccati operators is not sufficient to ensure that as the 
approximation order increases, the optimal cost |i, and a corresponding sequence of optimal actuator 
locations 7, converge. Since the cost is the norm of the Riccati operator, uniform convergence of the 
operators is required. That is, 


lim ||II,Py — || =0, 
N= OO. 


is needed in order to use approximations in determining optimal actuator location. The first point to 
consider is that since IT, has finite rank, II must be a compact operator in order for uniform convergence 
to occur, regardless of the choice of approximation method. Since the solution to an ARE is not always 
compact, it is not possible to compute the optimal actuator location for every problem. 
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Example 67.7: [8] 


Consider any A, B,C, such that A* = —A and C, = B*. Then II =/ is a solution to the ARE 
A* T+ TIA — T1BB*I1+C*C, =0. 


The identity / is not compact on any infinite-dimensional Hilbert space. 


Example 67.8: 


This example is a generalization of [6, Example 1]. On the Hilbert space Z = IR x H where H is any 
infinite-dimensional Hilbert space, define 


ee Erle bel a=[% ee | 


where M is a bounded operator on H. The solution to the ARE 


A*T1 + TIA — T1BB*11+C7'C, =0 


1 0 
l= s 
[ow | 
This operator is not compact if M is not a compact operator; for instance if M = |. This example is 


particularly interesting because A is a bounded operator and also generates an exponentially stable 
semigroup. 


Theorem 67.11: [31, Theorem 2.9,3.3] 


If B and C, are both compact operators, then the Riccati operator 1 is compact. Furthermore, if a sequence 
of approximations satisfy (A1), (Al*) and are uniformly stabilizable and detectable, then the minimal 
nonnegative solution 11, to (Equation 67.17) converges uniformly to the nonnegative solution I] to (Equa- 
tion 67.15): limn_soo ||TIn — H|| = 0. 


Thus, if B and C; are compact operators, guaranteeing compactness of the Riccati operator, then any 
approximation method satisfying the assumptions of Theorem 67.10 will lead to a convergent sequence of 
optimal actuator locations. A finite-dimensional input space guarantees that B is a compact operator; and 
similarly a finite-dimensional output space will guarantee that C; is compact, although these assumptions 
are not necessary. 

For an important class of problems the Riccati operator is compact, even if the observation operator 
C| is not compact. 


Definition 67.12: 


A semigroup S(t) is analytic if t > S(t) is analytic in some sector |argt| <0. 


Analytic semigroups have a number of interesting properties [34,37]. Recall that the solution S(t)z € 
D(A), t > 0, if z € D(A). IfS is an analytic semigroup, S(t)z € D(A) for all z € Z. Also, the eigenvalues 
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of the generator A of an analytic semigroup lie in a sector |arg\| < m — €, where € > 0. The heat equation 
and other parabolic PDEs lead to an analytic semigroup. Weakly damped wave and beam equations are 
not associated with analytic semigroups. 

If A generates an analytic semigroup, uniform convergence can be obtained without compactness of 
the state weight C). The result [22, Theorem. 4.1], applies to operators B and C, that may be unbounded. 
It is stated below for bounded B and Cy. 


Theorem 67.12: 


Let A generate an analytic semigroup S(t) with ||S(t)|| < Me’ and define A= (wI — A) for w > wo. 
Assume that the system (A, B, C,) has the following properties: 


1. (A,B) is stabilizable and (A, C,) is detectable. 

2. Either C¥C, > rl,r > 0, or for some F € L(Y, Z) such that A — FC, generates an exponentially stable 
semigroup, A-!FC, is compact. 

3. Either B*A~' is compact or there exists a compact operator K € £L(Z,U) such that A — BK generates 
an exponentially stable semigroup. 


Assume the following list of properties for the approximation scheme, where y is any number 0 <y <1: 


1. Forallz € Z, ||Pnz—z\| > 0. 
2. The approximations are uniformly analytic. That is, for some € > 0 


Moe(@otst 


|Apes"*|| < a > 0, OS 0=1. 


3. For some s and y independent of n,0 <y <1, 
a. ||A71— Az Pll < ™, 
b. ||B*z — B*Pyz|| < MnO) |[z|| pias, z € D(A*). 
Then limy- 00 ||TInPn — TI || = 0. 


The above conditions on the approximation scheme imply assumptions (Al) and (A1*) as well as 
uniform stabilizability and detectability. 

Provided that IT, converges to II in operator norm at each actuator location, the sequence of optimal 
actuator locations for the approximations converges to the correct optimal location. 


Theorem 67.13: [31, Theorem 3.5] 


Let Q be a closed and bounded set in RN. Assume that B(r), r € Q, is compact and such that for any ro € Q, 


Jim |1B(r) — B(r0)|| = 0. 


Assume also that (An, By(r), Cin) is a family of uniformly stabilizable and detectable approximations 
satisfying (A1) and (A1*) such that for each r, 


lim ||T,(r) — T1(r)|| = 0. 
n> CO 
Let ? be the optimal actuator location for (A, B(r), C1) with optimal cost \i; that is 


fi = inf ||T1(r)|] = || 1(7)|| 
reQ 
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FIGURE 67.11 Optimal actuator location and performance for different approximations of the viscously damped 
beam with weights C; = [J 0], R = 1. Although the output space is infinite-dimensional, C is a compact operator. 
This implies uniform convergence of the Riccati operators and thus convergence of both the optimal actuator locations 
7, and optimal costs jin. 


Defining similarly 7, in, it follows that 


Example 67.9: Viscously Damped Beam, cont. 


Consider the same viscously damped beam and control problem as in Example 67.6, except that 
now instead of trying to minimize the norm of the entire state, C; = /, we consider only the posi- 
tion. Choose the weight C; = [/ 0], where / here indicates the mapping from H3(0, 1) into £(0, 1). 
Although the semigroup is not analytic, both B and C; are compact operators on Z. Using the same 
modal approximations as before, we obtain convergence of the approximating optimal performance 
and the actuator locations. This is illustrated in Figure 67.11. 


Example 67.10: Diffusion (Eg. 67.1 cont.) 


Oz Oz 
SE aa Ore 0<x<1, 
|x ie 
br (x) = : 
0, Ix-rl>= 
2 
Oz te) 
—(0,t) = 0, —(1,t)=0 
5 (0, t) 5 (1,t) 


We wish to determine the best location r of the actuator to minimize 


lee) 1 
su20)= | 1000 f Iz(x, t)|2 dx + |u(t)|? dt 
0 0 
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FIGURE 67.12 Optimal actuator location and performance for approximations of the of the diffusion equation, 
C; = V 10001, R = 1. Since the semigroup is analytic, uniform convergence of II, to II is obtained, even for a 
noncompact Cj such as used here. This leads to convergence of the optimal performance and of a corresponding 
sequence of actuator locations. 


with respect to the worst possible initial condition. This means that we want to minimize ||IT(r)||, 
where IT solves the ARE with C; = /1000/ and R = 1. Note that C is not a compact operator. How- 
ever, A= 0? /dx? with domain D(A) = {z € H2(0, 1)|z’(0) = z’(1) = 0} generates an analytic semi- 
group on £3(0, 1). Defining Zp to be the span of the first n eigenfunctions and defining the corre- 
sponding Galerkin approximation as in Example 67.4 leads to an approximation that satisfies the 
assumptions of Theorem 67.12 [22] and so limn-—s oo || Mn(r) — N(/)|| = 0 for each location r. Conver- 
gence of the optimal performance and of a corresponding sequence of actuator locations is shown 
in Figure 67.12. 

Figure 67.13 illustrates that this optimal actuator location problem is nonconvex. We are only 
guaranteed to have convergence of a sequence of optimal actuator locations, not every sequence. 


The discussion in this section has so far been concerned only with state feedback. However, in general, 
the full state z is not available and a measurement 


y(t) = Coz(t), (67.18) 


where C2 € £(Z, W) and W is a finite-dimensional Hilbert space, is used to estimate the state. 

As for finite-dimensional systems, we construct an estimate of the state. Choose some F € L(W, Z) so 
that A — FC) generates an exponentially stable semigroup. This can be done, for instance, by solving a 
Riccati equation dual to that for control: F = WOR where for some B, € £(Z, V), V a Hilbert space 
and R, € L(W, W) with R, > 0, 


(DA* + AD +B, Bi — UCHR,'C)d)z=0, Vz € D(A*). (67.19) 


The estimator is 


Ze(t) = Aze(t) + Bu(t) + F(y(t) — Coze(t)). 


Stability of A — FC2 guarantees that lims_, oo ||Ze(t) — z(t)|| = 0. 
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FIGURE 67.13 The cost ||II(r)|| as a function of r for the heat equation, C = V 1000], is not a convex function. 


The controller is formed by using the state estimate as input to a state feedback controller. As explained 
in Section 67.3, this leads to the controller 


Z(t) = (A— FCa)z-(t)—[F  Ble(t), 


(67.20) 
yc(t) = —Kz,(t), 


where e(t) = r(t) — y(t) and 


si = y(t) me! C, 0 
y(t) = be = z(t) + ; u(t). 
As for finite-dimensional systems, if A— FC, and A— BK each generate an exponentially stable semi- 
group, then the above controller stabilizes the infinite-dimensional system (Equations 67.2 and 67.18) 
[8, Section 5.3]. 
The controller (Equation 67.20) is infinite-dimensional. A finite-dimensional approximation to this 


controller can be calculated using a finite-dimensional approximation (Ay, By, Cz) to the original system 
(A, B, Cy). Consider F, = D,C* R>! , where X,, solves the ARE 


2n-~e 
DnA* + An Un t+ BinBt, — UnC3,Rp 'Cond = 0. (67.21) 


Results for convergence of solutions &, and the operators F,, follow from arguments dual to those for 
convergence of solutions to the control Riccati equation 67.17. 


Theorem 67.14: 


Assume that assumptions (A1) and (A1*) hold for approximations of (4. [B Bi] ; el) and that the 
2 


approximations are also uniformly stabilizable and uniformly detectable. Then the operators Ky, F, obtained 
by solving the Riccati equations 67.17 and 67.21 respectively converge to the operators K and F obtained 
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by solving Equations 67.15 and 67.19. Furthermore, the sequence Ky, uniformly stabilizes (An, By) and the 
sequence F,, is uniformly detectable for (An, Can). 


Defining 


7(t) = b I = ie 2(t) + "| u(t) 


and letting e(t) = r — y, the finite-dimensional controller is 


Z(t) = (An = FyCon)Zc(t) 4 [Fn Bal e(t), 


(67.22) 
yc(t) = —Ky z(t). 


It follows from Theorem 67.8 that the sequence of controllers (Equation 67.22) converge in the gap 
topology to Equation 67.20, and that they stabilize the original system for large enough n. Furthermore, 
the corresponding closed-loop systems converge. 


I 


67.5 


co Control 


Many applications involve an unknown and uncontrolled disturbance d(t). An important objective of 
controller design in these situations is to reduce the system’s response to the disturbance. The system 
(Equations 67.2 and 67.5) become 

dz 


<, = Aalt) + Bult) +Dd(e), 2(0) =0 (67.23) 


with cost 
y(t) = Cyz(t) + Eyu(t). (67.24) 


Since we are interested in reducing the response to the disturbance, the initial condition z(0) is set to 
zero. We assume that d(t) € L2(0, 00; V), where V is a Hilbert space and that D € L(V, Z) is a compact 
operator. (This last assumption follows automatically if V is finite-dimensional.) The measured signal, or 
input to the controller is 

y2(t) = Coz(t) + End(t) (67.25) 


The operator C2 € £(Z, W) for some finite-dimensional Hilbert space W and E) € L(V, W). 
Let G denote the transfer function from d to y; and let H denote the controller transfer function: 


u(s) = H(s)¥2(s). 
The map from the disturbance d to the cost y; is 
Jr = Ci(sI — A) 1(Dd + Bit) 
= C(sI — A)~!(Dd + BH»). 


Using Equations 67.23 and 67.25 to eliminate y2, and defining 


Gi1(s) = Ci(sI — A)~'D, Gy2(s) = Ci(sI — A)~'B+ Ej, 
Gai (s) = Co(sI— A)"'D+ Ep, Gy2(s) = Ca(sI — A)~!B, 


we obtain the transfer function 
F(G, H) = Gii(s) + Gi2(s)H(s)(I — Gao(s)H(s))~ Gai (s) 


from the disturbance d to the cost y1. 
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The controller design problem is to find, for given y > 0, a stabilizing controller H so that 


[ Ineo < f y\|d(t)|I? dt. 
0 0 


If such a controller is found, the controlled system will then have L2-gain less than y. 


Definition 67.13: 


The system (Equations 67.23 through 67.25) is stabilizable with attenuation y if there is a stabilizing 
controller H so that 
IlF(G; A) loo < ¥- 


To simplify the formulae, we make the assumptions that 


Filcy. Ay) =[0° <4); B Je-[7]. (67.26) 


With these simplifying assumptions, the cost y; has norm 


/ Ino a= [ |Cyz(t)||? + |lu(t)|I* dt 
0 0 


which is the LQ cost (Equation 67.14) with normalized control weight R = I. The difference is that here 
we are considering the effect of the disturbance d on the cost, instead of the initial condition z(0). Also, 
the problem formulation (Equations 67.23 through 67.25) can include robustness and other performance 
constraints. For details, see, for example [30,41]. 

We assume throughout that (A, D) and (A, B) are stabilizable and that (A, C;) and (A, C2) are detectable. 
These assumptions ensure that an internally stabilizing controller exists; and that internal and external 
stability are equivalent for the closed loop if the controller realization is stabilizable and detectable. 

Consider first the full information case: 


Oe ee = E x(t) + H d(t). (67.27) 


An important characteristic of H-disturbance attenuation is that, in general, a system is not stabiliz- 
able with attenuation y for every y. However, if it is stabilizable with attenuation y, the attenuation (in 
the full-information case) can be achieved with constant state-feedback. 


Definition 67.14: 


The state feedback K € L(Z,U) is y-admissible if A-BK generates an exponentially stable semigroup and 
the feedback u(t) = —Kz(t) is such that y-attenuation is achieved. 


Theorem 67.15: [4,20] 


Assume that (A, B) is stabilizable and (A, C;) is detectable. For y > 0, the following are equivalent: 


¢ The full-information system (Equations 67.23, 67.24, and 67.27) is stabilizable with disturbance 
attenuation y. 
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¢ There exists a nonnegative, self-adjoint operator X on Z such that for allz € D(A), 
1 
(an+na+n (<:p0° - 28") n+cjc,)2=0 (67.28) 
¥ 


and A — BB*T1 + 32 DD* iat DDN generates an exponentially stable semigroup on Z. 


In this case, K = B*T] is a y-admissible state feedback. 


Note that as y — oo, the Riccati equation for the LQ problem is obtained. Since the operator Riccati 
equation 67.28 cannot be solved, a finite-dimensional approximation to the original infinite-dimensional 
system is used to approximate the exact control K = B*II. As in previous sections, let Z, be a finite- 
dimensional subspace of Z and P, the orthogonal projection of Z onto Z,. Consider a sequence of 
operators Ay, € L(Zy, Zn), By = PyB, Dn = PyD, and Ci, = Ci|z,,. Assumptions similar to those used for 
LQ control are required. 


Theorem 67.16: [16, Theorem 2.5, Corollary 2.6] 


Assume a sequence of approximations satisfy (A1),(A1*), (An; Bn) are uniformly stabilizable, and (An, Cin) 
are uniformly detectable. Assume that the original problem is stabilizable with attenuation y. For sufficiently 
large n the Riccati equation 


1 
AST TMyAg Ty ( [2D = BB Ty + C*,Cin = 0, (67.29) 


has a nonnegative, self-adjoint solution I1,. For such n 


¢ There exist positive constants M, and w, such that the semigroup S,2(t) generated by Ay+ 
J Dx Dj Tn — Bn Ba Un satisfies ||Sna(t)l| < Mie". 

¢ Ky, =(Bn)*l, is a y-admissible state feedback for the approximating system and there exists 
Mp, 2 > 0 such that the semigroup Snx(t) generated by Ay, + ByKy satisfies ||Snx(t)|| < Moe °?". 


Moreover, for allz € Z, TyPnz > Mz asn— 00 and Ky = (Bn)* Wn converges to K = B3 M1 in norm. For 


n sufficiently large, K,Pn is a y-admissible state feedback for the infinite-dimensional system. 


The optimal disturbance attenuation problem is to find 
y=infy 


over all y such that Equations 67.23, 67.24, and 67.27 are stabilizable with attenuation y. Let /, indicate 
the corresponding optimal disturbance attenuation for the approximating problems. Theorem 67.16 
implies that lim sup,_,.. Yn < Y but in fact convergence of the optimal disturbance attenuation can 
be shown. 


Corollary 67.1: [16, Theorem 2.8] 


With the same assumptions as Theorem 67.16, it follows that 


lim Yn= Vy. 


n—> Ooo 
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TABLE 67.1 Physical Constants 


E 2.1 10!! N/m? 
I 1.2x 107!° m4 
p 3.0 kg/m 

cy 0.0010 Ns/m2 
Cd 0.010 Ns /m? 

L 7.00 m 

Tp 39.0 kg m? 

Od 0.12 1/m 


Example 67.11: Flexible Slewing Beam 


Consider a Euler-Bernoulli beam clamped at one end and free at other end. Let w(x, t) denote the 
deflection of the beam from its rigid body motion at time t and position x. The deflection can be 
controlled by applying a torque at the clamped end (x = 0). We assume that the hub inertia /}, is 
much larger than the beam inertia, so that, letting 6(t) indicate the rotation angle, u(t) = I, A(t) isa 
reasonable approximation to the applied torque. The disturbance d(t) induces a uniformly distributed 
load pgd(t). The values of the physical parameters used in the simulations are listed in Table 67.1. Use 
of the Kelvin-Voigt model for damping leads to the following description of the beam vibrations: 


O2w Ow 2 O2w O7w ox 
El ] = t d(t), O<x<L. 
Fr tag + S| Dee ago et ee 3 
The boundary conditions are 
w(o,t)=0, 0, =0, 
Ox 


O2w O3w O3w O*w 
E/ —— +cgl—.— =0, E| —~- +cgl—~— =0. 
ax2 | 4 a ax3 | 7 ax3at 
x=L X=L 
Let z(t) = (w(., t), ow, t)), H¢(0, L) be the closed linear subspace of H2(0,L) defined by 
o dw 
H¢(0,L) = }w € H*(0,L) : w(0) = a 0) =0}. 
With the state space Z = H¢(0,L) x £7(0,L),a state-space formulation of the above PDE problem is 


oat) = Az(t) + Bu(t) + Dd(t), 


where 
0 0 | 
B=|x |, D=| pg], A=| Et d4 cgi d*+ ow |: 
Ih p o dx4 po dx4 0 


with, defining M = El 56 + cql 451y,A has domain 


D(A) ={(b, W) EX: WEH¢(0,L); Me H2(0,L) with M(L) = om) = 0}. 


The operators B and D are clearly bounded operators from R to Z. 
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Suppose the objective of the controller design is to reduce the effect of disturbances on the tip 
position: 
y(t) = Cyz(t) = wiL, t). 
Sobolev’s Inequality implies that evaluation at a point is bounded on H¢(0,L) and so C, is bounded 
from Z toR. 
Define the bilinear form on H¢(0, L) x H¢(0, L) 


LE q2 d2 
ol, y) = [ = £780) 55 V0) de 
Define V = H;(0,L) x H¢(0,L) and define a(-,-) on V x V by 
abr, br), (2, ba) = —9(1 42) + 011 +S Wa, tba) + Wis V2) 


for (;, Wj) € V, i= 1,2. Then A can be defined by 
(Ay,Z)p*.p =—aly,z), fory, zeEV. 


The form a(-,-) satisfies inequality (Equation 67.7) and also (Equation 67.8) with k <0 and so the 
operator A generates an exponentially stable semigroup on Z. 

Let Hn Cc H¢(0,L) be a sequence of finite-dimensional subspaces formed by the span of n standard 
finite-element cubic B—spline approximations [33]. The approximating generator Ay on Zn = Hy x 
Hn is defined by the Galerkin approximation 


(AnYn:Zn) = —@(Yn.Zn), VZn, Yn € Zn. 


For all  € H¢(0,L) there exists a sequence on € Hn with limn—oo [lon — >| H;(0,L) = 0 [33]. It follows 
then from Theorem 67.7 and exponential stability of the original system that (A1) is satisfied and 
that the approximations are uniformly exponentially stabilizable (trivially, by the zero operator). The 
adjoint of A can be defined through a(z, y) and (A1*) and uniform detectability also follow. Thus, 
Theorem 67.16 applies and convergence of the approximating feedback operators is obtained. 

The corresponding series of finite-dimensional Riccati equation 67 29 were solved with y = 2.3. 
Figure 67.14 compares the open- and closed-loop responses w(L, t) to a disturbance consisting of 
a 100 second pulse, for the approximation with 10 elements. The feedback controller leads to a 
closed-loop system which is able to almost entirely reject this disturbance. Figure 67.15 compares 
the open- and closed-loop responses to the periodic disturbance sin(wt), where w is the first resonant 
frequency: w = min; |Im(X;(A109)|. The resonance in the open loop is not present in the closed loop. 

Figure 67.16 displays the convergence of the feedback gains predicted by Theorem 67.16. Since 
Zn is a product space, the first and second components of the gains are displayed separately as 
displacement and velocity gains, respectively. 


In general, of course, the full state is not measured and the measured output is described by Equa- 
tion 67.25. 


Theorem 67.17: [4,20] 


The system (Equations 67.23 through 67.25) is stabilizable with attenuation y > 0 if and only if the following 
two conditions are satisfied: 


1. There exists a nonnegative self-adjoint operator I] on Z satisfying the Riccati equation 
1 
(4*11+4+1 (<p0" - BB) M+C*C\)z=0, Vze D(A), (67.30) 
Y 


such that A+ (3; DD* — BB*)II generates an exponentially stable semigroup on Z; 
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FIGURE 67.14 Hp. -state feedback for flexible slewing beam. Open-(..) and closed-loop (-) responses to a distur- 
bance d(t) = 1,t < 100 s: 10 elements. 


2. Define A= A+ 2 DDT and K = B*I1. There exists a nonnegative self-adjoint operator © on X 
satisfying the Riccati equation 


ries i ~[{1 7 

(AZ+ ZA* +E (SKK = C3) X+DD*)z=0, Vze D(A*), (67.31) 
Y 

such that A+ 4 (5K*K am C3C2) generates an exponentially stable semigroup on X. 
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FIGURE 67.15 Hp -state feedback for flexible slewing beam. Open-(..) and closed-loop (—) responses to a distur- 
bance d(t) = sin(wt) : 10 elements. 
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FIGURE 67.16 Hoo-state feedback for flexible slewing beam. Feedback gains for 2 elements *, 4 elements ....., 6 
elements _._., 8 elements __, and 10 elements, __ are plotted. As predicted by the theory, the feedback operators are 
converging. 


Moreover, if both conditions are satisfied, define F = >y, On! and A, =A4 Zz DD* — BK — FC). The 
controller with state-space description 


Z(t) = Acz(t) + Fya(t), (67.32) 
u(t) = —Kz,(t) 


stabilizes the system (Equations 67.23 through 67.25) with attenuation y. 


Condition (1) is simply the Riccati equation to be solved for the full-information state feedback con- 
troller. Condition (2) leads to an estimate of the state of the system controlled by Kz(t). Unlike LQ control, 
the design of the state feedback and the estimator is coupled. Condition (2) above is more often written 
as the following two equivalent conditions: 


a. There exists a nonnegative, self-adjoint operator X on X satisfying the Riccati equation 
1 
(az +ZA*+z (Gera = C302) I +pp") z=0, Vze D(A*) (67.33) 
¥ 


such that A + (Js Cy C; — C} Cz) generates an exponentially stable semigroup on Z 
b. r(ST1) < y? where r indicates the spectral radius. 


In the presence of condition (1) in Theorem 67.17, condition (2) is equivalent to conditions (a) and (b). 
Also 5 = (I— 2 yn) ly = DU - eld). The advantage of replacing condition (2) by conditions 
(a) and (b) is numerical. The Riccati equation in (2) is coupled to the solution of (1) whereas the Riccati 
equation in (a) is independent of the solution of (1). This theoretical result has been extended to a class 
of control systems with unbounded control and observation operators [20]. 
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For bounded control and observation operators, a complete approximation theory exists. Define a 
sequence of approximations on finite-dimensional spaces Zy, as for the full information case, with the 
addition of Cz, = C2|z,. 

Strong convergence of solutions X,, to Riccati equations approximating Equation 67.33 will follow from 
Theorem 67.16 and a straightforward duality argument if (A1) and (A1*) hold, along with assumptions 
on uniform stabilizability of (A, D) and uniform detectability of (A, C2). However, strong convergence 
of both II, — II and X, — & does not imply convergence (or even existence) of the inverse operator 
(I - ye enln) so we cannot show controller convergence. Convergence of the solution x, to the 
estimation Riccati equation 67.31 can be proven. 


Theorem 67.18: [27, Theorem 3.5] 


Assume that (A1) and (A1*) hold, that (An, Dy) are uniformly stabilizable, and that (Ay, Con) are uniformly 
detectable. Let y > 0 be such that the infinite-dimensional problem is stabilizable. Let N be large enough 
that approximations to the full-information problem are stabilizable with attenuation y, and let V1, indicate 
the solution to the ARE (Equation 67.29) for n > N. Define K, = B* I, and An =Ant eDnDaUn : 

For sufficiently large n the Riccati equation 


ee Zin, e 1 as 
Age tea eS, (SKK - C$,Can) En +D,D* =0 (67.34) 


has a nonnegative, self-adjoint solution Xn. For such n there exist positive constants M3 and w3 such that 
the semigroup Sy2(t) generated by Ay, + Zz Unk Kn — UnCj,,Con satisfies ||Sy2(t)|| < M3e—°3 © Moreover, 


for eachz € Z, Pn Z—> Dzasn— coand Fy, = DnC} converges toF= UC in norm. 


Defining Acy = An 2 D,D*,X — Bn Ky — FyCon, Theorems 67.16 and 67.18 imply convergence of the 
controllers 
Z(t) = AcnZc(t) + Frny2(t), (67.35) 
u(t) = —Knz-(t) 


to the infinite-dimensional controller (Equation 67.32) in the gap topology. The same assumptions imply 
convergence of the plants which leads to the following result. 


Theorem 67.19: [27, Theorem 3.6] 


Let y be such that the infinite-dimensional system is stabilizable with attenuation y. Assume that (A1) and 
(A1*) hold, that (A, B) and (A, D) are uniformly stabilizable, and that (A, C,) and (A, C2) are uniformly 
detectable. Then the finite-dimensional controllers (Equation 67.35) converge in the gap topology to the 
infinite-dimensional controller (Equation 67.32). For sufficiently large n, the finite-dimensional controllers 
(Equation 67.35) stabilize the infinite-dimensional system and provide y-attenuation. 


Convergence of the optimal attenuation also holds for the output feedback problem. 


Corollary 67.2: [27, Theorem 3.7] 


Let ) indicate the optimal disturbance attenuation for the output-feedback problem (Equations 67.23 
through 67.25), and similarly indicate the optimal attenuation for the approximating problems by Y,. With 
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the same assumptions as for Theorem 67.19, 


lim Yn=/- 


n— oo 


67.6 Summary 


For most practical systems, an approximation to the PDE must be used in controller design. Similarly, con- 
trol for delay-differential equations often proceeds by using an approximation, although delay-differential 
equations were not discussed directly in this chapter. This has the advantage of making available the vast 
array of tools available for design of controllers for finite-dimensional systems. Since the underlying model 
is infinite-dimensional, this process is not entirely straightforward. However, there are a number of tools 
and techniques available for satisfactory controller design. This chapter has presented an overview of the 
main issues surrounding controller design for these systems. The key point is that the controller must 
control the original system. Sufficient conditions for satisfactory LQ controller design and Hoo-controller 
design were presented. Uniform stabilizability and detectability, along with convergence of the adjoint 
systems, are assumptions not required in simulation but key to obtaining satisfactory performance of 
a controller designed using a finite-dimensional approximation. There are results guaranteeing these 
assumptions for many problems. However, for problems where these assumptions are not known and 
proof is not feasible, the approximating systems should be checked numerically for uniform stabilizability 
and detectability. One test is to verify that the sequence of controlled systems is uniformly exponentially 
stable. 

This chapter discussed only systems with bounded control and observation operators B and C. Intro- 
ducing a better model for an actuator sometimes converts a control system with an unbounded control 
operator to a more complex model with a bounded operator, and similarly for sensor models, e.g. [17,43]. 
For some systems, however, the most natural model leads to unbounded operators. There are considerably 
fewer results for controller design for these systems. However, results do exist for LQ control of parabolic 
problems, such as diffusion [2,22]. 

Once a suitable approximation scheme, and controller synthesis technique, has been chosen, the 
problem of computation remains. The primary difficulty is solving the large Riccati equations that arise 
in LQ and Hoo-controller design. For problems where an approximation of suitable accuracy is of relatively 
low order (less than about a hundred), direct methods can be used to solve the Riccati equations. However, 
for larger problems, an iterative method is required. Probably the most popular method for the Riccati 
equations that arise in LQ control is the Newton-Kleinman method [21] and its variants—see, for 
example [10,32]. This method is guaranteed to converge, and has quadratic convergence. However, 
calculation of an Hoo-controller corresponds to calculation of a saddle point, not a minimum as in the 
case of LQ control. Suitable methods for design of Hoo-controllers for large-scale systems is an open 
problem at the present time. 
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68.1 Dynamic Elastic Beam Models 


Consider the deformation of a thin, initially curved beam of length @ and constant cross-section of area 
A, which, in its undeformed reference configuration, occupies the region 


Q = {x =: r9(x1) + x2€2(x1) + x3€3(x1)| x1 € [0, 2], 


(x2, x3) == x2€2(x1) + x3€3(x1) € A} 


68-1 
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where ro : [0, 2] > R? is a smooth function representing the centerline, or the reference line, of the beam 
at rest. The orthonormal triads e;(-), e2(-), e3(-) are chosen as smooth functions of x; so that e; is the 
direction of the tangent vector to the centerline, i.e., e1(x,) = (dro/dx1)(x1), and e2(x;), e3(x1) span the 
orthogonal cross section at x;. The meanings of the variables x; are as follows: x; denotes arc length along 
the undeformed centerline, and x2 and x3 denote lengths along lines orthogonal to the reference line. The 
set Q can then be viewed as obtained by translating the reference curve ro(x;) to the position x2e2 + x3e3 
within the cross-section perpendicular to the tangent of ro. 

Ata given time f, let R(x, x2, x3, t) denote the position vector after deformation to the particle which is 
at r(x), x2, x3) in the reference configuration. We introduce the displacement vector by V := R—r. The 
position vector R(x, 0, 0, t) to the deformed reference line x2 = x3 = 0 is denoted by Ro. Accordingly, the 
displacement vector of a particle on the reference line is W := V(x1, 0,0) = Ro — ro. The position vector 
R may be approximated to first order by 


R=Ro4+ Eo + *3Es, 


where E; are the tangents at R with respect to xj, respectively. Note, however, that the triad E; is not 
necessarily orthogonal, due to shearing. 

The deformation of r(-) into R(-., t) will be considered as a succession of two motions: (1) a rotation 
carrying the triad e;(x,) to an intermediate orthonormal triad é;(x1, t), followed by (2) a deformation 
carrying €;(x1,t) into the nonorthogonal triad E;(x1, t). The two triads é; and E; then differ on account 
of a strain € to be specified below. We choose to orient the intermediate (right-handed) triad é;, which 
serves as a moving orthonormal reference frame, so that 


E; = (FE) -€))é; =|Ei|@1,  €2- E3 = €3 - Ep. 
A strain é, related to the deformation carrying the triad é; to the triad Ej, is defined by 


€)-E, =:1+81, 1: Ey =: 28:2, 
@)-E, =:1+822, €, - Es =: 283, 


@3-E3 =:1+833, @)-E3 =: €3. 


The remaining strains are defined by requiring the symmetry & = é;j. If distortion of the planar cross 
sections is neglected, then €22 © 33 © €23 © 0. The normal N to the cross section is then N = E2 x E3 = 
ey = 2821 = 2831é3. 

Let ©; denote the angles associated with the orthogonal transformation carrying the orthonormal basis 
e; into é;, whereas the rotation of e; into E; is represented by the angles 3; (dextral mutual rotations). Up 
to quadratic approximations we obtain 


01 =O), 
Uy = O72 + 2&3] + 201 €), 
03 = ©O3 -— 2&2] + 20163}. 


These angles are interpreted as the global rotations. It is obvious from these relations that the shear strains 
vanish if, and only if, the angles ©;, 0; coincide, i = 1, 2,3, or, what is the same, if the normal N to the 
cross section coincides with E). This is what is known as the Euler-Bernoulli hypothesis. 

To complete the representation of the reference strains in terms of the angles above and the displace- 
ments Wj; := W-e;, we compute up to quadratic approximations in all rotations and linear approxi- 
mations in all strains &;. To this end we introduce curvatures and twist for the undeformed reference 
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configuration by Frénet-type formulae 
K2=e2-€1,1, K3=@3-€1,1, T=€3-€2,1. 


(The index separated by a comma indicates a partial derivative with respect to the corresponding variable 
x;.) The reference strains are then approximated by 


4 1 

€11 = Wy,1 — K2 W2 — k3 W3 + 5 Wa + k3W, + tW)? + (Wo +k2W1 — tW3)”), 
_ 1 

£21 = 5 (Wat — 03 + K2 Wy — tW3 +01 (02 + W3,1 + k3W 1 + tW2)), 

7 1 

3] = 5 (Wat +92 +k3Wy + TW. + 91093 — Wo) — Kp W] + TW3)), 


whereas the approximate bending strains are 


Ko = 03,1 +301 + th, 
K3 = —V21 — k2) + 103, 


t= O11 _ K202 _ K303. 


The approximations given above comprise the theory of rods with infinitesimal strains and moderate 
rotations; see Wempner [27]. 


68.2 The Equations of Motion 


Under the assumptions of the previous section, the total strain (or potential) energy is given by 


L 
EA_ st <a: dias SEs. Nel 
u= | | Spats + 26a +a) + 5 + > 4 se | as, 


where E, G, Ino, 133, I = Ino + 133 are Young’s modulus, the shear modulus, and moments of the cross 
section, respectively. The kinetic energy is given by 


e 
K = \|Ri|?dx, 
0 


where = d/dt. Controls may be introduced as distributed, pointwise, or boundary forces and couples 
(F, M, f, m) through the total work 


53 
w= | (F-W+M-9)dx+ 5S (f-W+m-9). 
0 x=0,,0 


Controls may also be introduced in geometric boundary conditions, but we shall not do so here. Typically, 
a beam will be rigidly clamped at one end (say at x; = 0) and simply supported or free at the other end, 
x, =. If the space of test functions Vo := {f € H 10, £)\|f (0) = 0} is introduced, the requirement that 
the end x; = 0 be rigidly clamped is mathematically realized by the “geometric” constraints Wj, 3; € Vo. 
If x; = 0 is simply supported, rather than clamped, the appropriate geometric boundary conditions are 
W; € Vo for i= 2 andi = 3 only. 

Let 


T 
ee} [K(t) + W(t) —U(f)] dt 
0 


be the Lagrangian. Then, by Hamilton’s principle (see, for example [26]), the dynamics of the deformation 
satisfy the stationarity conditions Cw = 0, 8£y = 0, the variations being in Vo. The equations of motion 
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then follow by integration by parts, collecting terms, etc., in the usual manner. Due to space limitations, 
neither the most general beam model nor the entirety of all meaningful boundary conditions can be 
described here. Rather, a partial list of beam models which, in part, have been studied in the literature and 
which can easily be extracted from the formulas above, will be provided. We focus on the typical situation 
of a beam which is clamped at x) = 0 and free (resp., controlled) at x; = &. 


68.2.1 Initially Straight and Untwisted Linear Shearable 3-Dimensional 
Beams 


68.2.1.1 Equations of Motion 


moW, =[EAW\)’ (longitudinal motion) 
mg W> = [GA(W3 — 93)]' (lateral motion) 
moW3 = [GA(W3 + 9) (vertical motion) 
x (68.1) 
mot = [GI9) (torsional motion) 
mos. = [EI3395]' — GA(W5 + 92), (shear around é2) 
moo3 = [Eln2¥,]' + GA(W; — 93) (shear around é3) 
where’ = d/dx). 
68.2.1.2 Boundary Conditions 
The geometric boundary conditions are 
W;(0)=0, 9;(0)=0, i=1,2,3. (68.2) 
The dynamical boundary conditions are 
EAW( (2) =f» 
GA(W3 — 33)(€) = fr, 
GA(W3 + 32)(2) =fa, 
(68.3) 


GI8{ (0) =m, 
E33 05(£) = ma, 


Eln23 (£) = ms. 


68.2.1.3 Initial Conditions 


Wi(-,t = 0) = Wio(-), Wil-,t = 0) = Wal-), 
0(-,f=0)=0, 0(.,t=0)=0, i=1,2,3. 
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68.2.2 Initially Straight and Untwisted, Nonshearable Nonlinear 
3-Dimensional Beams 


68.2.2.1 Equations of Motion 
These are comprised of four equations which describe the longitudinal, lateral, vertical, and torsional 


motions, respectively. 


7 1 1 4 
mW, = E (w; + 5 Wa) + (ws) | F 


A, Arh my / 1 1)2 1 1)2 / ; 
mg W2 — [polo2 W5]' + [Eh2Wy]" = | | EACW; + <(W3)° + <(W5)°) WS) 
2 2 (68.4) 
r i 1 1 : 
mo W3 — [P0133 W3]' + [EI33 W3]” = | (zac + 5 (Wa) + sows) | : 
pol3, = [GI94]. 
68.2.2.2 Boundary Conditions 
The geometric boundary conditions are 
W;(0)=0, i=1,2,3, 
(68.5) 
W3(0) = W5(0) = 91(0) =0, 
while the dynamical boundary conditions are 
1 
EA (™: + ((W3)? + (ws) ®=fi 
El33W3(£)=—m2, Elo, Wy (£) = m3, 
1 e 
[Bacw - ((W5)? «ws | W3(4) — [Eh2Wy)' (4) + pola W3(0) = fr, (68.6) 


[Bac CA «ws | W3(0) — [E3 Wy’ (2) + pol33 W3(0) = —fs» 
GI9 (0) =m 


68.2.2.3 Initial Conditions 


Wi(-,t = 0) = Wyo(-), Wil-,t = 0) = Wa(-), 
b(,t=0)=0, (,f=0)=0, i=1,2,3. 


68.2.3 Nonlinear Planar, Shearable Straight Beams 
68.2.3.1 Equations of Motion 


The three equations which describe the longitudinal, vertical, and shear motions, respectively, are 
1 / 
mohW, = |Emw; te w9| ; 


*: 1 : 68.7 
mo W3 = [Gh(92 + W3)) + |Emw; + swows| ; ey) 


0013392 = El33 05 Gh(s, 4 W3). 
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68.2.3.2 Boundary Conditions 
The geometric boundary conditions are 
W,(0)=0, i=1,3, 2(0)=0, (68.8) 


while the dynamical boundary conditions are 
1 
Eh | + 3 | (O=fi, 


Childs + WO + Ew n swows | (=f, (68.9) 


E3395, (€) =m. 
68.2.3.3 Initial Conditions 


Wi(,t=0)= Woo), Wil,t=0)= Wal), i= 1,3, 
02(,f=0) = 20,  b2(-,f = 0) = V1. 


Remark 68.1 


The model in this section is attributed to Hirschhorn and Reiss [7]. If the longitudinal motion is neglected 
and the quadratic term in W3 is averaged over the interval [0, 2], the model then reduces to a Woinowski- 
Krieger type. The corresponding partial differential equations are quasi-linear and hard to handle. If, 
however, one replaces @2 by 32 in the expression for the strain €), (which is justified for small strains), 
then a semilinear partial differential equation with a cubic nonlinearity in 02 is obtained. 


68.2.4 Planar, Nonshearable Nonlinear Beam 


The equations of motion for the longitudinal and vertical motions, respectively, are 


‘ 1 ; 
mW = jEacw +swy).. 
' ; (68.10) 
mo W3 — [pol33 W3]' + [El33 W4]" = | (zac aa 5(wsy) ws] : 


We dispense with displaying the boundary and initial conditions for this and subsequent models, as 
those can be immediately deduced from the previous ones. This model has been derived in Lagnese and 
Leugering [16]. 

The models above easily reduce to the classical beam equations as follows. We first concentrate on the 
nonshearable beams. 


68.2.5 The Rayleigh Beam Model 


Here the longitudinal motion is not coupled to the remaining motions. The equation for vertical motion is 
mo W3 — [pol33 W3]' + [El3 W411" = 0. (68.11) 
68.2.6 The Euler—Bernoulli Beam Model 
This is obtained by ignoring the rotational inertia of cross sections in the Rayleigh model: 
mo W3 + [El33W3]"" = 0. (68.12) 


With regard to shearable beams, two systems are singled out. 
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68.2.7 The Bresse System 


This is a model for a planar, linear shearable beam with initial curvature involving couplings of longitu- 
dinal, vertical, and shear motions. 


mo W = [Eh(W{ — k3W3)]' — K3Gh(d2 + W5 + «3 Wi), 
mo W3 = [Gh(82 + W3 + «3 Wi)]' + K3EA[ Wy — «3 Ws], (68.13) 


pol3302 = Els30, — Gh(82 + Ws +k3W)1). 


This system was first introduced by Bresse [6]. 


68.2.8 The Timoshenko System 


This is the Bresse model for a straight beam («3 = 0), so that the longitudinal motion uncouples from the 
other two equations, which are 


mo W3 = [Gh(92 + W4)T | 
(68.14) 


po13302 = Elz3, — Gh(82 + W5). 


Remark 68.2 


The models above can be taken to be the basic beam models. In applications it is also necessary to account 
for damping and various other (local or non-local) effects due to internal variables, such as viscoelastic 
damping of Boltzmann (non-local in time) or Kelvin-Voigt (local in time) types, structural damping, so- 
called shear-diffusion or spatial hysteresis type damping; see Russell [24] for the latter. It is also possible 
to impose large (and usually fast) rigid motions on the beam. We refer to [9]. A comprehensive treatment 
of elastic frames composed of beams of the types discussed above is given in [17]. With respect to control 
applications, one should also mention the modeling of beams with piezoceramic actuators; see Banks 
et al. [3]. 


68.3 Exact Controllability 


68.3.1 Hyperbolic Systems 


Consider the models (Equations 68.1, 68.7, 68.13, and 68.14). We concentrate on the linear equations 
first. All of these models can be put into the form 


Mz = [K(z’ + Cz)]/ — C’K(z’ + Cz) +f, (68.15) 
z(0)=0, K(z'+Cz)(é)=u, (68.16) 
z(-, 0) =Z0, z(-, 0) =7), (68.17) 


with positive definite matrices M,K depending continuously on x. In particular, for the model (Equa- 
tion 68.1) 


z=(W,9)', M= diag(mo, mo, mo; pol, Pol33> Pol22) 
K = diag(EA, GA, GA, GI, E133, Elz), 
Co =—-1, C35=1, Cy =0 otherwise. 
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In the case of the Bresse beam, 


z= (Wj, W3,92)?, M = diag(mo, mo, 00133), 
K = diag(Eh, Gh, GA, El33), 


Cy2=—-K3, Co =kK3, C3 =1, Ci =0 otherwise. 


The Timoshenko system is obtained by setting k3 = 0. If C=0, Equation 68.15 reduces to the one- 
dimensional wave equation. 
We introduce the spaces 


H=17(0,4,R1), V={zeH'(0,4,R1): 2(0) =0}, (68.18) 


where q is the number of state variables and H*(0, £, R2) denotes the Sobolev space consisting of R4 valued 
functions defined on the interval (0, 2) whose distributional derivatives up to order k are in L7(0, 64). 
(The reader is referred to [1] for general information about Sobolev spaces.) We further introduce the 
energy forms 


1 ré 
(z,Z)yv = al K(z' + Cz) - (z’ + Cz) dx, 
0 
1 7 
(z, Z)q =5/ Mz-z dx. 
2 Jo 


Indeed, the norm induced by ||z|| = (z, zy) is equivalent to the usual Sobolev norm of H : given by 


e 1/2 
\Izl]1, = (/ (\2’ P+ 2?) dx) 
0 


Let zo € V, z; € H, f € L*(0, T, H), u € L*(0, T, R4), T > 0. It can be proven that a unique function 
z€ C(0,T,V)NC!(0, T, H) exists which satisfies Equations 68.15, 68.16, and 68.17 in the following weak 
sense: ; 

d = 
pz 2): On + (ZC), d)v = (M 'f(t), a +u(t)-o(, Voev, (68.19) 


and F 
(z(0), >) = (Zo, >), FAC) o)ulr=0 = (Z1, >) x. 


Because of space limitations, only boundary controls and constant coefficients will be considered. 
Distributed controls are easy to handle, while pointwise controls have much in common with boundary 
controls except for the liberty of their location. Thus we set f = 0 in Equation 68.15. The problem of exact 
controllability in its strongest sense can be formulated as follows: Given initial data Equation 68.17 and 
final data (zr0,z11) in V x H and given T > 0, find a control we L?(0, T, R2) so that the solution z of 
Equation 68.15 satisfies Equations 68.16, 68.17 and z(T) = z70,2z(T) = z711. 

It is, in principle, possible to solve the generalized eigenvalue problem 


d* a 
2 2 2 
—o=2(1-y—)o, 
ae (I-y ae )o 
$(0) = 40) = oD =" + 9/0 =0, 
and write the solution of Equations 68.15, 68.16, 68.17 using Fourier’s method of separation of variables. 
The solution together with its time derivative can then be evaluated at T and the control problem reduces 


to a trigonometric (or to a complex exponential) moment problem. The controllability requirement is 
then equivalent to the base properties of the underlying set of complex exponentials [exp(i,t)|k € Z,t € 
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(0, T)]. If that set constitutes a Riesz basis in its L”(0, T)-closure, then exact controllability is achieved. 
For conciseness, we do not pursue this approach here and, instead, refer to Krabs [12] for further reading. 
Rather, the approach we want to consider here, while equivalent to the former one, does not resort to the 
knowledge of eigenvalues and eigenelements. The controllability problem, as in finite dimensions, is a 
question of characterizing the image ofa linear map, the control-to-state map. Unlike in finite dimensions, 
however, it is not sufficient here to establish a uniqueness result for the homogeneous adjoint problem, 
that is, to establish injectivity of the adjoint of the control-to-state map. In infinite-dimensional systems 
this implies only that the control-to-state map has dense range, which in turn is referred to as approximate 
controllability. Rather, we need some additional information on the adjoint system, namely, uniformity 
in the sense that the adjoint map is uniformly injective with respect to all finite energy initial (final) 
conditions. In particular, given a bounded linear map L between Hilbert spaces X, Y, the range of L is 
all of Y if, and only if, L*, the adjoint of L, satisfies |||| < v||L*|| for some positive v and all  € Y. 
This inequality also implies that the right inverse L*(LL*)~! exists as a bounded operator (see the finite- 
dimensional analog, where ‘bounded’ is generic). It is clear that this implies norm-minimality of the 
controls constructed this way. This result extends to more general space setups. It turns out that an 
inequality like this is needed to assure that the set of complex exponentials is a Riesz base in its L?-closure. 
As will be seen shortly, such an inequality is achieved by nonstandard energy estimates, which constitute 
the basis for the so-called HUM method introduced by Lions. It is thus clear that this inequality is the 
crucial point in the study of exact controllability. 

In order to obtain such estimates, we consider smooth enough solutions ¢ of the homogeneous adjoint 
final value problem, 


Mo = [K(o’ + Co)’ — C7K($’ + Cd), (68.20) 
(0) =0, K(’+Co)(é) =0, (68.21) 
o(.T)=0, (-,T) = $1. (68.22) 


Let m be a smooth, positive, increasing function of x. Multiply Equation 68.20 by mo’, where m(-) is 
a smooth function in x, and integrate by parts over (x, t). After some calculus, we obtain the following 
crucial identity, valid for any sufficiently smooth solution of Equation 68.20: 


£ T T pe 
0= / mM} - (! + Co)|o dx — i m(x)e(x, t)|{_ dt + / [ m edxdt 
0 0 0 Jo 
T pe — T pe 
= / i mC! Md- bdxdt + / i mC! K(' + Co) - (’ + Co) dxdt, (68.23) 
0 Jo 0 Jo 
where p = fe mM} - (o’ + Co) dx and e(x, t) denotes the energy density given by 
j eae ae 
eS 5 Mer K($! + Co) - (o + Co)]. 
Set the total energy at time ft, 


e 
c= [ e(x, t) dx, (68.24) 
0 


and denote the norm of the matrix C by v. If we choose m in Equation 68.23 so that m’(x) — vm(x) > 
co > 0, Vx € (0, £), we obtain the estimates 


E T 
y lb(2, t)|*dt < €(0) <T / \d(2, t)|"dt, (68.25) 
0 0 


for some positive constants y(T), I(T), where T is sufficiently large (indeed, T > 2 “optical length of 
the beam” is sufficient). The second inequality in Equation 68.25 requires the multiplier above and some 
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estimation, whereas the first requires the multiplier m(x) = —1+ 2x/é, and is straightforwardly proved. 
These inequalities can also be obtained by the method of characteristics which, in addition, yields the 
smallest possible control time [17]. It is then shown that the norm of the adjoint to the control-to-state 
map (which takes the control u into the final values z(T),z(T) (for zero initial conditions)), applied to 
bo, 1, is exactly equal to fis |d(, t)|?dt. By the above argument, the original map is onto between the 
control space L?(0, T, R4) and the finite energy space V x H. 


Theorem 68.1: 


Let (Zo, 21); (ZT0,ZT1) be in V x H and T > 0 sufficiently large. Then a unique control u € L*(0, T, R4) 
exists, with minimal norm, so that z satisfies Equations 68.15, 68.16, 68.17 and z(T) = z70,2(T) = 271. 


Remark 68.3 


Controllability results for the fully nonlinear planar shearable beam Equation 68.7, and also for the 
Woinowski-Krieger-type approximation, are not known at present. The semilinear model is locally 
exactly controllable, using the implicit function theorem. The argument is quite similar to the one com- 
monly used in finite-dimensional control theory. 


68.3.2 Quasi-Hyperbolic Systems 


In this section we discuss the (linearized) nonshearable models with rotational inertia, namely, Equa- 
tions 68.4, 68.10, 68.11. We first discuss the linear subsystems. Observe that in that situation all equations 
decouple into wave equations governing the longitudinal and torsional motion and equations of the type 
of Equation 68.11. Hence it is sufficient to consider the latter. For simplicity, we restrict ourselves to 
constant coefficients. The system is then 


ohW — pIW” + EIW”” =f, 
W(0)=0, W’(0)=0, 


and 


EIW"(€)=m, (EIW™ — pIW’)(0) = u2, 


: (68.26) 
W(-,0)= Wo, W(-,0) = Wy. 


Setting y* := I/A and rescaling by t > t,//E, this system can be brought into a nondimensional form. 
We define spaces, 


H=[veH!(0,: v(0)=0], 
V=[veH*(0,4: v(0)=v'(0) =0), (68.27) 


and forms, 


£ 
(u,v) = i uvdx, (uv)q =(uv)+y?(uVv), 
0 
(uv)y =y?(u",v"). 


Let Wo € V, W) € H,andu € L’(0, T, R’), T > 0. It may be proved that there is a unique W € C(0, T, V)N 
C1(0, T, H) satisfying Equation 68.26 in an appropriate variational sense. 
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Remark 68.4 


The nonlinear models can be treated using the theory of nonlinear maximal monotone operators; see 
Lagnese and Leugering [16]. 

To produce an energy identity analogous to Equation 68.23, we multiply the first equation of Equa- 
tion 68.26 by xW’ — aW, where a > 0 is a free parameter, and then we integrate over (0, £) x (0, T). If 
we introduce the auxiliary functions p; = if W(xW’ —aW) dx and p2 = y” ie W'(xW’ —aWY dx, and 
P = 01 + (2, we find after some calculus 


err. ; 
0= o(T) ~ (0) ~ 5 i (We DP + PLW'(e, £7} at 
0 


T Ae 
Ea? / [W'(é, t))’dt + / (2 WwW” —yW’)(6, (CW! — aW)(Z, t)) dt 
0 0 


# ‘ 1 w2 2 1 (W’)? 4 2 3 (w")? dxdt 
PAN ar la el 


With the total energy now defined by 


1 
E(t) = sll W(t) ile + WWI} (68.28) 


this identity can now be used to derive the energy estimate, 
T . . 
x / HGH) +y bY Gdldt < 0) 
0 


T 
<1 i [52(6, 1) + 208 P(G.1)] dt, 
0 


for some positive constants m(T) and I(T), which is valid for sufficiently smooth solutions ¢ to the 
homogeneous system, Equation 68.26, and for sufficiently large T > 0 (again T is related to the “optical 
length,” i.e., to wave velocities). The first estimate is more standard and determines the regularity of 
the solutions. It is again a matter of calculating the control-to-state map and its adjoint. After some 
routine calculation, one verifies that the norm of the adjoint, applied to the final data for the backwards 
running homogeneous equation, coincides with the time integral in the energy estimate. This leads to 
the exact controllability of the system, Equation 68.26, in the space V x H, with V and H as defined in 
Equation 68.27, using controls u = (uy, uz) € L*(0, T, R2). 


Remark 68.5 


As in [18], for y > 0, the controllability results for Rayleigh beams carry over to the corresponding results 
for Euler-Bernoulli beams. It is however instructive and, in fact, much easier to establish controllability 
of the Euler-Bernoulli beam directly with control only in the shear force. 


68.3.3 The Euler—-Bernoulli Beam 
We consider the nondimensional form of Equation 68.12, namely, 
W + wl" = 0 
Ww(0)=W(0)=0, W"(O)=0, W"(O)=u (68.29) 
W(-,0)= Wo, W(,é)= Wi. 


We introduce the spaces 


H=17(0,4, V={ve H?|v(0) = v'(0) = 0} (68.30) 
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and the corresponding energy functional 
a oeeer: 
EO) = 5(IWOlla + IWOMly. (68.31) 


Again, we are going to use multipliers to establish energy identities. The usual choice is m(x) = x. Upon 
introducing p = 7 xW W’dx and multiplying the first equation by mW’, followed by integration by parts, 
we obtain 


rt re. 3 rT ré ert. T 
o(T) — p(0) + = i i W? dxdt + al i (W")dxdt = al Weare | W’(é, t)u(t) dt. 
2Jo Jo 2Jo Jo 2 Jo 0 
(68.32) 
Using this identity for the homogeneous system solved by , we obtain the energy estimates 


wh T 
xf (6, t) dt < €(0) < nf °(é, t) dt 
0 0 


where again m(T) and I1(T) depend on T > 0, with T sufficiently large. 
One way to obtain the adjoint control-to-state-map is to consider 


L 
al {Wo+ Wo" }dx = —u(t)o(é, t), 
dt Jo 


for W as above and solving the backwards running adjoint equation (i.e., Equation 68.29 with final 
conditions 79 and 71). Integrating with respect to time over (0, T) yields 


T 
(Lr(u), (o7T0, 671))vxH = -| u(t)o(F, t) dt. 
0 


The same argument as above yields the conclusion of exact controllability of the system (Equation 68.29), 
in the space V x H, where V and H are defined in Equation 68.30, using controls u € L?(0,T, R). 


Remark 68.6 


It may be shown that the control time T for the Euler-Bernoulli system can actually be taken arbitrarily 
small. That is typical for this kind of model (Petrovskii type systems) and is closely related to the absence of 
a uniform wave speed. The reader is referred to the survey article [15] for general background information 
on controllability and stabilizability of beams and plates. 


68.4 Stabilizability 


We proceed to establish uniform exponential decay for the solutions of the various beam models by lin- 
ear feedback controls at the boundary x = @. There is much current work on nonlinear and constrained 
feedback laws. However, the results are usually very technical, and, therefore, do not seem suitable for 
reproduction in these notes. In the linear case it is known that for time reversible systems, exact con- 
trollability is equivalent to uniform exponential stabilizability. In contrast to the finite-dimensional case, 
however, we have to distinguish between various concepts of controllability, such as exact, spectral, or 
approximate controllability. Accordingly, we have to distinguish between different concepts of stabiliz- 
ability, as uniform exponential decay is substantially different from nonuniform decay. Because of space 
limitations, we do not dwell on the relation between controllability, stabilizability, and even observability. 
The procedure we follow is based on Lyapunov functions, and is the same in all of the models. Once 
again the energy identities, Equations 68.23, 68.28, and 68.32 are crucial. We take the hyperbolic case as 
an exemplar and outline the procedure in that case. 
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68.4.1 Hyperbolic Systems 


Apply Equation 68.23 to solve Equation 68.15 (with f = 0) and Equation 68.16 with the control u in 
Equation 68.16 replaced by a linear feedback law u(t) = —kx(6, t), k > 0. Recall that p = ie mMz.:- (z' + 
Cz) dx(t). Then using Equation 68.23, 


b < yl2(¢,t)|? — co€(t), 
where € is given by Equation 68.24. Therefore, introducing the function F(t) := E(t) + €p(t) one finds 
Fe < E(t) + yela(G t)|? — eco€(t). 


However, €(t) = —k|z(é¢, t)|*, and therefore the boundary term can be compensated for by choosing € 
sufficiently small. This results in the estimate F(t) < —cy€€(t) for some cy > 0, which in turn implies 


t 
Fat)+o f E(s)ds < F(0). 
0 


It also straightforward to see that F (f) satisfies 
TeE(t) < F(t) < NeE(t). 


The latter implies is E(s)ds < (1/X)E(0), with } = I, /(cye) and t > 0. One defines 1(t) : a hie E(s)ds 
and obtains a differential inequality 7+» <0. A standard Gronwall argument implies y(t) < 


exp(—At)n(0), that is, 
jy £(s)ds < HPN) 50), 
t 


Now, because €(t) is nonincreasing, 


“GLAS | [ He | eas as < SP) M4) 69) 


and this, together with the choice t = 1/2, gives 
E(t) < eexp(—dt)E(0),  Vt>t. (68.33) 


Hence, we have the following result. 


Theorem 68.2: 


Let V and H be given by Equation 68.18. Given initial data zp, 2 in V x H, the solution to the closed-loop 
system, Equations 68.15 through 68.17, with u(t) = —kz(¢,t), satisfies 


E(t) < Mexp(—wt)&(0), t>0, (68.34) 


for some positive constants M and w. 


Remark 68.7 


The linear feedback law can be replaced by a monotone nonlinear feedback law with certain growth 
conditions. The corresponding energy estimates, however, are beyond the scope of these notes. Ultimately, 
the differential inequality above is to be replaced by a nonlinear one. The exponential decay has then (in 
general) to be replaced by an algebraic decay, see [17]. The decay rate can be optimized using “hyperbolic 
estimates” as in [17]. Also the dependence on the feedback parameter can be made explicit. 
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68.4.2 Quasi-Hyperbolic Systems 
We consider the problem of Equation 68.26 with feedback controls, 
ui(t)=—kW'(Gt), w(t)=kW(64), (68.35) 


with positive feedback gains k; and kz. The identity Equation 68.23 is used to calculate the derivative 
of the function p(t). By following the same procedure as above, we obtain the decay estimate Equa- 
tion 68.34 for the closed-loop system, Equations 68.26 and 68.35, where the energy functional € is given 
by Equation 68.28. 


Remark 68.8 


One can show algebraic decay for certain monotone nonlinear feedbacks. In addition, the nonlinear 
system, Equation 68.10, exhibits those decay rates as well; see Lagnese and Leugering [16]. 


68.4.3 The Euler—-Bernoulli Beam 


Here we consider the system Equation 68.29 and close the loop by setting u(t) = —kW(4,t), k > 0. By 
utilizing the estimate Equation 68.32 and proceeding in much the same way as above, the decay estimate 
Equation 68.34 can be established for the closed-loop system, where € is given by Equation 68.31. 


68.5 Dynamic Elastic Plate Models 


Let be a bounded, open, connected set in R? with a Lipschitz continuous boundary consisting of a 
finite number of smooth curves. Consider a deformable three-dimensional body which, in equilibrium, 
occupies the region 

[(x1,%25%3): (x12) €Q, [x3] < h/2). (68.36) 
When the quantity h is very small compared to the diameter of Q, the body is referred to as a thin plate 
of uniform thickness h and the planar region, 


[(x1, X250) : (x1, x2) € Q] 


is its reference surface. 

Two-dimensional mathematical models describing the deformation of the three-dimensional body 
Equation 68.36 are obtained by relating the displacement vector associated with the deformation of each 
point within the body to certain state variables defined on the reference surface. Many such models are 
available; three are briefly described below. 


68.5.1 Linear Models 


Let W(x1, x2, x3, t) denote the displacement vector at time t of the material point located at (x1, x2, x3), and 
let w(x1, x2, t) denote the displacement vector of the material point located at (x1, x2, 0) in the reference 
surface. Further, let n(x1, x2, t) be the unit-normal vector to the deformed reference surface at the point 
(x1, X2,0) + w(x1, x2, t). The direction of n is chosen so that n-k > 0, where i,j,k is the natural basis 
for R>. 


68.5.1.1 Kirchhoff Model 


The basic kinematic assumption of this model is 
W(x1, 25.3, t) = W(Xx1, x2, t) + x3(n(x1, x2, t) —k), (68.37) 


which means that a filament in its equilibrium position, orthogonal to the reference surface, remains 
straight, unstretched, and orthogonal to the deformed reference surface. It is further assumed that the 
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material is linearly elastic (Hookean), homogeneous and isotropic, that the transverse normal stress 
is small compared to the remaining stresses, and that the strains and the normal vector n are well- 
approximated by their linear approximations. Write w = w)i-+ w2j + w3k. Under the assumptions above, 
there is no coupling between the in-plane displacements w1, w2 and the transverse displacement w3 := w. 
The former components satisfy the partial differential equations of linear plane elasticity and the latter 
satisfies the equation 
ow Ow , 
LA +DA?w=F, (68.38) 
at 

where A = 0? /Ox? + 07 /Ox} is the harmonic operator in R’, p is the mass density per unit of reference 
volume, Ip = ph? /12 is the polar moment of inertia, D is the modulus of flexural rigidity, and F is the 
transverse component of an applied force distributed over (2. The “standard” Kirchhoff plate equation is 
obtained by omitting the term I, A(0*w/0t*), which accounts for the rotational inertia of cross sections, 
from Equation 68.38. 


68.5.1.2 Reissner—Mindlin System 


The basic kinematic assumption of this model is 
W(x1, X2; X35 t) = w(x, X2, t) + x3U (x1, x2, t), (68.39) 


where |U+k| = 1. Equation 68.39 means that a filament in its equilibrium position, orthogonal to 
the reference surface, remains straight and unstretched but not necessarily orthogonal to the deformed 
reference surface. Write 

U = Ujit U2j+ U3k, u= Uji+ Upj. 


In the linear approximation, U3 = 0 so that the state variables of the problem are w and u. The latter 
variable accounts for transverse shearing of cross sections. It is further assumed that the material is 
homogeneous and Hookean. The stress-strain relations assume that the material is isotropic in directions 
parallel to the reference surface but may have different material properties in the transverse direction. 
As in the previous case, there is no coupling between w , w2, and the remaining state variables in the 
linear approximations. The equations of motion satisfied by w3 := w and u, referred to as the Reissner or 
Reissner-Mindlin system, may be written 


@2 
ph — Ghdiv (u+ Vw) =F, and 
(68.40) 
ou WB 
op 7 pi o(u) + Gh(u+ Vw) =C, 


where Gh is the shear modulus and o(u) = (o;;(u)) is the stress tensor associated with u, i.e., 


2 
2pr es 
oi (u) a 2Le;j(u) + ret 2B eck (UW), j= 1,2. 


ej(u) denotes the linearized strain 


(u) 1 (OU; i OU; 
ey(u) = = > 
: BACON: SON 


and w are the Lamé parameters of the material, 


3 
div(u+Vw)=V-(u+ Vw), divo(u) = Sy aay 
zi Ox; 


and C = Cji+ C)j is a distributed force couple. 
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68.5.2 A Nonlinear Model: The von Karman System 


Unlike the two previous models, this is a “large deflection” model. It is obtained under the same assump- 
tions as the Kirchhoff model except for the linearization of the strain tensor. Rather, in the general strain 
tensor, 

1 (2m oni) 1 4 OW, OW, 


| 
T > 
2 Ox; Ox; 2 kel Ox} Ox; 


ej(W) = 


the quadratic terms involving W3 are retained, an assumption formally justified if the planar strains 
are small relative to the transverse strains. The result is a nonlinear plate model in which the in-plane 
components of displacement w; and w2 are coupled to the transverse displacement w3 := w. Under some 
further simplifying assumptions, w; and w2 may be replaced by a single function G, called an Airy stress 
function, related to the in-plane stresses. The resulting nonlinear equations for w and G are 


Ow Ow 


2 
I,A ae DA‘w — [w, G] = F, 


(68.41) 
5 Eh 
A*G+ 5 lwwl =0, 
where E is Young’s modulus and where 


POP Py ao 4 > Oy 
Ox? Ox} Ax? Ax} Ax Oxy Ox1Ox2” 


Lb, wv] = 


One may observe that [w, w]/2 is the Gaussian curvature of the deformed reference surface x3 = w(x), x2). 
The “standard” dynamic von Karman plate system is obtained by setting I, = 0 in Equation 68.41. 


Remark 68.9 


For a derivation of various plate models, including thermoelastic plates and viscoelastic plates, see [18]. 
For studies of junction conditions between two or more interconnected (not necessarily co-planar) elastic 
plates, the reader is referred to the monographs [17,21] and references therein. 


68.5.3 Boundary Conditions 


Let I’ denote the boundary of Q. The boundary conditions are of two types: geometric conditions, that 
constrain the geometry of the deformation at the boundary, and mechanical (or dynamic) conditions, 
that represent the balance of linear and angular momenta at the boundary. 


68.5.3.1 Boundary Conditions for the Reissner-Mindlin System 


Geometric conditions are given by 
w=w, u=u onl, t>0. (68.42) 


The case w = u = 0 corresponds to a rigidly clamped boundary. 
The mechanical boundary conditions are given by 


Ghv - (u+ Vw) =f, 
43 (68.43) 


—o(u)v=c onT,t>0, 
12 


where v is the unit exterior pointing normal vector to the boundary of Q, ¢ = c,i+ cj is a boundary force 
couple and f is the transverse component of an applied force distributed over the boundary. The problem 
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consisting of the system of Equations 68.40 and boundary conditions Equations 68.42 or 68.43, together 
with the initial conditions 


Ow 
w= w?, ae 
(68.44) 
9 Ou 1 
u=u, —e=u_ att=0, 
Ot 


has a unique solution if the data of the problem is sufficiently regular. The same is true if the boundary 
conditions are Equation 68.42 on one part of F and Equation 68.43 on the remaining part, or if they 
consist of the first (resp., second) of the two expressions in Equation 68.42 and the second (resp., first) of 
the two expressions in Equation 68.43. 

68.5.3.2 Boundary Conditions for the Kirchhoff Model 

The geometric boundary conditions are 

Ow _ 


—z=-—v-u onl,t>0, (68.45) 
Ov 


and the mechanical boundary conditions may be written 


(68.46) 


2 (1,2 paw) 2 r-otvwpl = 2-4 
p w ee See f. 
68.5.3.3 Boundary Conditions for the von Karman System 
The state variables w and G are not coupled in the boundary conditions. The geometric and mechanical 
boundary conditions for w are those of the Kirchhoff model. The boundary conditions satisfied by G are 
OG 

=0 


G=0, —=0. 
on) 


(68.47) 


These arise if there are no in-plane applied forces along the boundary. 


68.6 Controllability of Dynamic Plates 


In the models discussed in the last section, some, or all, of the applied forces and moments F, C,f,c, 
and the geometric data w, u, may be considered as controls which must be chosen in order to affect the 
transient behavior of the solution in some specified manner. These controls may either be open loop, 
or closed loop. Open-loop controls are usually associated with problems of controllability, which is that 
of steering the solution to, or nearly to, a specified state at a specified time. Closed-loop controls are 
usually associated with problems of stabilizability, that is, of asymptotically driving the solution towards 
an equilibrium state of the system. 

In fact, for infinite-dimensional systems of which the above plate models are representative, there are 
various related but distinct concepts of controllability (spectral, approximate, exact) and of stabilizability 
(weak, strong, uniform), distinctions which disappear in finite-dimensional approximations of these 
models (see [2, Chapter 4]). Stabilizability problems will be discussed in the next section. With regard 
to controllability, exact controllability is the most stringent requirement because it requires a complete 
description of the configuration space (reachable set) of the solution. This is equivalent to steering any 
initial state of the system to any other permissible state within a specified interval of time. The notion of 
spectral controllability involves exactly controlling the span of any set of finitely many of the eigenmodes of 
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the system. Approximate controllability involves steering an arbitrary initial state to a given, but arbitrary, 
neighborhood of a desired configuration within a specified time. 

Among the possible controls, distinctions are made between distributed controls such as F and C, 
which are distributed over all or a portion of the face of the plate, and boundary controls, such as f,c, w, U, 
which are distributed over all or a portion of the edge of the plate. Within the class of boundary controls, a 
further distinction is made between mechanical controls, f and c, and geometric controls w and u. Because 
mechanical controls correspond to forces and moments, they are, in principle, physically implementable; 
these are the only types of controls which will be considered here. In addition, only boundary control 
problems will be considered in detail; however, some remarks regarding distributed control problems 
will be provided. 


68.6.1 Controllability of Kirchhoff Plates 


Assume that ! = Ty) UT, where lp and I are disjoint, relatively open subsets of C with [; 4%. The 
problem under consideration consists of the partial differential equation 68.38, boundary conditions 
Equation 68.45 on I'9, boundary conditions Equation 68.46 on I), and initial conditions 


w(x, 0) = w°(x), as 0)=w(x), xeEQ. (68.48) 


In this system, the distributed force F, the geometric quantities w,u, and the initial data (w°, w!) are 
assumed as given data, while f,c are the controls, chosen from a certain class C of admissible controls. The 
configuration space, or the reachable set, at time T is 


Rr = {(w(T), w(T)) : (f,€) € C}, 


where, for example, w(T) stands for the function [w(x, T) : x € Q] and where w = Ow/0t. If z denotes 
the solution of the uncontrolled problem, ie., the solution with f = 0 and c = 0, then 


Rr = Rr @ {lz(T), 2(T)]}, 


where R¥4. denotes the configuration space when all of the given data are zero. Therefore, to study the 
reachable set it may be assumed without loss of generality that the data F, w, u, w°, w! vanish. The problem 


under consideration is, therefore, 


Ow Ow 


Deas 
h aE eA Ge tDArw=0, (68.49) 
w=0, dee a To,t >0, (68.50) 
ov 
6) aw hb a (6) 
I DA -o(Vw)v] = —(t- 
x ( ° Op w) ae lS Be td. 
(68.51) 
3 
Ovi S —v-c only,t>0, 
Ow 
w(x, 0) = Fraane =0, xEQ. (68.52) 


If w is a solution of Equation 68.49, its kinetic energy at time t is 


1 
K(t)= 5 | (oni? + 119%?) a0, 
Q 
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where the quantities in the integrand are evaluated at time t. The strain energy of this solution at time t is 
given by 
1B S 
Ut)=s5 d y: o,(Vw)ei(Vw) dQ. 
j= 


A pair of functions (wo, w;) defined on Q is called a finite energy pair if 


2 
ey oij(Vwo)ey(V wo) dQ < oo, 
Q 


ij=l 
/ (phw7 +1,|Vwi|?) dQ < 00. 
Q 


A solution w of Equation 68.49 is called a finite energy solution if [w(t), w(t)] is a finite energy pair for 
each t > 0 and is continuous with respect to t into the space of finite energy pairs. This means that the 
solution has finite kinetic and strain energies at each instant which vary continuously in time. 

Many choices of the control space C are possible, each of which will lead to different configuration 
space R¥.. One requirement on the choice of C is that the solution w corresponding to given input, f,c be 
reasonably well behaved. Another is that the choice of C lead to a sufficiently rich configuration space. For 
the problem under consideration, it is very difficult to determine the precise relation between the control 
and configuration spaces. For example, there is no simple characterization of those inputs for which 
the corresponding solution has finite energy at each instant. On the other hand, when standard control 
spaces with simple structure, such as L” spaces, are utilized, the regularity properties of the solution are, 
in general, difficult to determine. (This is in contrast to the situation which occurs in the analogous 
boundary control problem for Rayleigh beams, where it is known that finite energy solutions correspond 
exactly to inputs which are L? in time.) 

In order to make the ideas precise, it is necessary to introduce certain function spaces based on the 
energy functionals K and U/. Let L?(Q) denote the space of square integrable functions defined on , and 
let H() be the Sobolev space consisting of functions in L?(Q) whose derivatives up to order k (in the 
sense of distributions) belong to L?(Q). Let 


H ={ve H}(Q): vr, = 0}. 
The quantity 
1/2 
Illa = ( i (phv? +1pl¥v?) de) 
Q 


defines a Hilbert norm on H which is equivalent to the standard induced H!(Q) norm. Similarly, define 


Ov 
V={veH: veH(Q), —| =0}. 

V|r, 

The quantity 
3 1/2 
lIvilv = (/ (Feucreycen) an) 
Q 

defines a seminorm on V. In fact, as a consequence of Korn’s lemma, || - || y is actually a norm equivalent 


to the standard induced H*() norm whenever Ip # Y. Such will be assumed in what follows to simplify 
the discussion. The Hilbert space V is dense in H and the injection V +> H is compact. Let H be identified 
with its dual space and let V* denote the dual space of V. Then H C V* with compact injection. A finite 
energy solution of Equations 68.49 through 68.51 is characterized by the statements w(t) € V, w(t) € H 
for each t, and the mapping t +> (w(t), w(f)) is continuous into the space V x H. The space V x H is 
sometimes referred to as finite energy space. 
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Write ¢ = c,i+ cj. In order to assure that the configuration space is sufficiently rich, the control space 
is chosen as 
C=({f.0: felix O,T)l,  €Vl1 x (0,T)I}. (68.53) 


The penalty for this simple choice is that the corresponding solution, which may be defined in a certain 
weak sense and is unique, is not necessarily a finite energy solution. In fact, it can be shown by variational 
methods that, if w is the solution of Equations 68.49 through 68.52 corresponding to an input (f,c) €C, 
then w(t) € H, w(t) € V* and the mapping t > [w(t), w(t)] is continuous into H x V*. (A more refined 
analysis of the regularity of the solution may be found in [20].) 


68.6.1.1 Approximate Controllability 


The system (Equations 68.49 through 68.52) is called approximately controllable at time T if Rf. is dense 
in H x V*. To study this problem, introduce the control-to-state map Cr defined by 


Cr: Cte HxV*, Cr(f,c) =[w(T), w(T)]. 


Then the system, Equations 68.49 through 68.52, is approximately controllable at time T exactly when 
range(Cr) is dense in H x V*. The linear operator Cr is bounded, so, therefore, is its dual operator 
Cy: Hx V++C. Thus, proving the approximate controllability of Equations 68.49 through 68.52 is 
equivalent to showing that 


(b.@°)EHXV, Cp(o1, 6°) = 0 = (1, °) =0. 
The quantity C%(', 6°) may be explicitly calculated (see, [14]). It is given by the trace 
Cro", 6°) = (6, VOIP, x<(0,r)> 


where 6 is the solution of the final value problem 


ao ao 
h— —I,A—~ +DA*o=0, 68.54 
pha Aaa +DA'> ( ) 
fe} 
b=0, —=0 only,0<t<T, (68.55) 
Ov 
O a h? a 
I DA -o(V =0, 
dv ( an °) 12 ore VON 
(68.56) 
3 
pO jv=0 only,0<t<T, 
0 Ob 1 
o(x,T)=o, Bi (xT)=o0, xEQ. (68.57) 


Therefore, the system, Equations 68.49 through 68.52, is approximately controllable if the only solution 
of Equations 68.54 through 68.57, which also satisfies 


blr, x(o,7) =9 Volr,x«o,r) = 9, (68.58) 


is the trivial solution. However, the boundary conditions, Equations 68.56 and 68.58, together, imply 
that satisfies Cauchy data on I x (0, T), that is, @ and its derivatives up to order three vanish on 
Tl; x (0, T). If T is large enough, a general uniqueness theorem (Holmgren’s theorem) then implies that 
@ = 0 in Q x (0, T). This implies approximate controllability in time T. 
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Theorem 68.3: 


There is a To > 0 so that the system Equations 68.49 through 68.52 is approximately controllable in time 
T >To. 


Remark 68.10 


The optimal time Tp depends on the material parameters and the geometry of Q and I). If Q is convex, 


then Ty = 2,/I,/D d(&,1 ), where 


d(Q,1\) = sup inf |x — y|. 
xeQyeli 


Remark 68.11 


(Distributed control.) Consider the problem of approximate controllability using a distributed control 
rather than boundary controls. Let w be a nonempty, open subset of Q and let the control space be 


C=[F: Fe L’(Qx(0,T)), F=0inQ\o. 


Consider the system consisting of Equation 68.38 and (for example) the homogeneous boundary condi- 
tions, 
Ow 
w= —=0 onl,t>0. 
ov 


Assume that the initial data is zero and let 
RY. = {(w(T), W(T)) : F EC}. 


For any input F taken from C the corresponding solution may be shown to be a finite energy solution. 
Let H and V be defined as above with [9 = I. The control-to-state map Cr : Fre (w(T), w(T)) maps C 
boundedly into V x H and its dual is given by 


Cr, ©") = lox (0,7) 
where 6 is the solution of Equation 68.54 with final data, Equation 68.57, and boundary conditions 


0 
o=0, at =0 onT,0<t<T. (68.59) 
Av) 


If b|ox (0,7) = O and T is sufficiently large, an application of Holmgren’s theorem gives > = 0 in Q x (0, T) 
and, therefore, the system is approximately controllable in time T. When Q is convex, the optimal control 


time is To = 2,/1p/D d(Q, w). 


68.6.1.2 Exact Controllability 


Again consider the system Equations 68.49 through 68.52 with the control space given by Equation 68.53. 
If D is a subspace in H x V*, the system is exactly controllable to D at time T if D C RY}. The exact 
controllability problem, in the strictest sense, consists of explicitly identifying RY. or, in a less restricted 
sense, of explicitly identifying dense subspaces D of H x V* contained in R¥. 

To obtain useful explicit information about RY. it is necessary to restrict the geometry of M9 and I’). It 
is assumed that there is a point xo € R? so that 


(x —x09):v <0, xETo. (68.60) 


Condition (Equation 68.60) is a “nontrapping” assumption of the sort found in early work on scattering 
of waves from a reflecting obstacle. Without some such restriction on Tg, the results described below 
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would not be valid. It is further assumed that 
ToNT, =9. (68.61) 


This is a technical assumption needed to assure that solutions of the uncontrolled problem have adequate 
regularity up to the boundary (cf. Remark 68.13 below). 


Theorem 68.4: 


Under the assumptions (Equations 68.60 and 68.61), there is a Tp > 0 so that 
RrDVXH (68.62) 
if T > To. 
The inclusion stated in Equation 68.62 may be stated in equivalent form in terms of the control-to-state 


mapping Cr. In fact, let 
Co = {(f,¢) EC: Cr(f,c) eVx H}, 


and consider the restriction of Cr to Co. This is a closed, densely defined linear operator from Cp into 
V x H. The inclusion (Equation 68.62) is the same as the assertion C7(Co) = V x H, the same as proving 
that the dual of the operator has a bounded inverse from V* x H to Co, that is, 


T 
10°. Rv se ff @? +1v0r) 40, (68.63) 
0 Ey 
where is the solution of Equations 68.54 through 68.57. For large T, the “observability estimate” 
(Equation 68.63) was proved in [18] under the additional geometric assumption 
(x—xo):v>0, xe). (68.64) 


However, this hypothesis may be removed by application of the results of [20]. 


Remark 68.12 


It is likely that the optimal control time To for exact controllability is the same as that for approximate 
controllability, but that has not been proved. 


Remark 68.13 


The hypothesis (Equation 68.61) may be replaced by the assumption that the sets ') and Ty meet in a 
strictly convex angle (measured in the interior of 2). 


Remark 68.14 


The conclusion of Theorem 68.4 is false if the space of controls is restricted to finite-dimensional con- 
trollers of the form, 


N 
c(x,t) =) ai(x)ei(2), 
i=1 


N 
f@HN= > pif, «eT, t>0, 


i=1 


where aj, B; are given L?(T,) functions and Cj, fi are L*(0, T) controls, i= 1,...,N(see [25]). 
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Remark 68.15 


Given a desired final state (wo, wi) € V x H, there are many ways of constructing a control pair (f,c) 
so that w(T) = wo and w(T) = wy. The unique control of minimum L(P, x (0, T)) norm may be con- 
structed as follows. Set ©; =I) x (0, T). Let be the solution of Equations 68.54 through 68.57 and let 
w be the solution of Equations 68.49 through 68.52 with 


f=ls,, and c=—Voly,. (68.65) 
Then [w(T), w(T)] depends on (°, $'). A linear mapping A is defined by setting 
A(°, 61) = [w(T), —w(T)]. 


The inequality (Equation 68.63) may be used to show that, for any (wo, w1) € V x H, the pair (°, 6!) may 
be chosen so that A(°, 6!) = (wi, —wo). The argument is based on the calculation (using integrations 
by parts) 


T 
o= | [ e¢onie — asso + Daw) doce, 
0 Q 
= i [ohiv(T)$° + Ip Vi(T)- V9 — phw(T)! —1,Vw(T)- Vo! ] dQ — ke (2 + |Vol?) dB, 
Q X1 


that is, 
(A649) 8" 6 Dinar = f ($7 + |Vol*) dd. (68.66) 


According to Equation 68.63, for T large enough, the right-hand side of Equation 68.66 defines a Hilbert 
norm ||(°, o')||z and a corresponding Hilbert space F which is the completion of sufficiently smooth 
pairs (°, o') with respect to || - |p. The identity (Equation 68.66) shows that A is exactly the Riesz 
isomorphism of F onto its dual space F*. Because (wo, w1) € geass precisely when (w1, —wo) € range(A), 
it follows that 
Rit = [(wo, w1) : (1, —Wo) € F*]. 

The inequality (Equation 68.63) implies that H x V* D F. Therefore H x V C F*, whichis the conclusion 
of Theorem 68.4. If (wo, wi) € V x H, then the minimum norm control is given by Equation 68.65, where 


(°, o!) = A7!(w 1, —wo). This procedure for constructing the minimum norm control is the basis of the 
Hilbert Uniqueness Method introduced in [22,23]. 


Remark 68.16 


In the situation where J, = 0 in Equations 68.49 and 68.51, the only change is that H = L?(Q) rather than 
the space defined above and the optimal control time is known to be Tp = 0, i-e., exact controllability 
holds in arbitrarily short time (cf. [28]). 


Remark 68.17 


If distributed controls are used rather than boundary controls as in Remark 68.11, Equation 68.62 is not 
true, in general, but is valid if is a neighborhood of T. 


68.6.2 Controllability of the Reissner-Mindlin System 


The controllability properties of the Reissner-Mindlin system are similar to those of the Kirchhoff system. 
Asin the last subsection, only the boundary control problem is considered in detail. Again, we work within 
the context of L? controls and choose Equation 68.53 as the control space. As above, it may be assumed 
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without losing generality that the data of the problem, F, C, w, u, w°, w, u°, u! vanish. The problem under 
consideration is, therefore, 


e2 
ph — Ghdiv (u+ Vw) =0, 
(68.67) 
TW seals Cha Vin =0 
PDE D lv Oo(u u w)=UVU, 
w=0, u=0 onl g,t>0, (68.68) 
Ghv - (u+ Vw) =f, 
3 (68.69) 
pes onlT),t>0, 
Ow 
,0) = —— (x, 0) = 0, 
w(x, 0) BE (x, 0) 
(6) 
u(x, 0) = (x, 0) =0 in Q. (68.70) 


For convenience, it is assumed that 'p 4 9. 
Set w = u+ wk and introduce the configuration space (reachable set) at time T for this problem by 


Ry = {[w(T),w(T)]}:  (f,¢) €C}. 


To describe R4, certain function spaces based on the kinetic and strain energy functionals of the above 
problem must be introduced. Let 


Po~weniteoke 


1/2 
Iva = (/ (ohw? + Ip?) de) : 
Q 


V={veH: uj,weH'(Q), vis, = 0}, 


ut+wk: uj;,w €L?(Q)], 


1/2 
Be ‘ ; 
Ivllv = [ a5 S> oi (u)eq(u) + Ghlut Vw? | dQ 
Q ij=l 
It is a consequence of Korn’s lemma and Ip ¥ & that || - ||v is a norm equivalent to the induced H 1(Q) 


norm. The space V is dense in H and the embedding V +> H is compact. A solution of Equations 68.67 
through 68.70 is a finite energy solution if w(t) € V, w(t) € H for each t, and the mapping t > (w(t), w(t)) 
is continuous into V x H. This means that the solution has finite kinetic and strain energies at each 
instant. As with the Kirchhoff model, solutions corresponding to inputs taken from C are not necessarily 
finite energy solutions. However, it is true that w(t) € H, w(t) € V* and the mapping t + [w(t), w(t)] is 
continuous into H x V*, where the concept of a solution is defined in an appropriate weak sense. 

The system, (Equations 68.67 through 68.70) is called approximately controllable if R4, is dense in 
H x V*. The exact controllability problem consists of explicitly identifying dense subspaces of H x V* 
contained in R¥.. 

With this setup, Theorem 68.3 and a slightly weaker version of Theorem 68.4 may be proved for 
the Reissner-Mindlin system. The proofs again consist of an examination of the control-to-state map 
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Cr: Ct> H x V* defined by Cr(f, c) = [w(T), w(T)]. The dual mapping C7. : H x V + C is given by 
Ci(0', &°) = Op, x07, 
where 
P=6+Vk, b= hit oj, 
P= M+ yk, b=6'4+W'k, 
and , th satisfy 


2, 
she — Ghdiv (6+ Vs) = 0, 
ot (68.71) 
op bh. , 
YV=0, 6=0 only,0<t<T, (68.72) 
Ghv - (b+ Vy) = 0, 
3 (68.73) 
po =0 onl),0<t<T, 
wen=w, Ven=vi 
: (68.74) 
(x, T) = 4°, se T)=6! ing. 


Approximate controllability amounts to showing that C7.(!, 6°) = 0 only when (!, ©°) = 0. However, 
if ® is the solution of Equations 68.71 through 68.74 and satisfies ®|r, .(o,r) = 0, then ® and its first 
derivatives vanish on T’; x (0, T). If T is large enough, Holmgren’s theorem then implies that 6 = 0. 

With regard to the exact controllability problem, to prove the inclusion, (Equation 68.62) for the 
Reissner system amounts to establishing the observability estimate (cf, Equation 68.63), 


T 
(0°, b!)|[Z,, ye < ef |@|-dTde. (68.75) 
0 Ty 


For sufficiently large T, this estimate has been proved in [18] under assumptions, Equations 68.60, 68.61, 
and 68.64. The following exact controllability result is a consequence of Equation 68.75. 


Theorem 68.5: 


Under assumptions Equations 68.60, 68.61, and 68.64, there is a Ty > 0 so that 
RE DVXxH 
if T > To. 


Remark 68.18 


If Q is convex, then the optimal control time for approximate controllability is 


To = 2 max(,/Ip/D, V 0/G) d(Q, 11). 


The optimal control time for exact controllability is probably the same, but this has not been proved. 
See, however, [10,11, Chapter 5]. Assumption, Equation 68.64, is probably unnecessary, but this has not 
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been established. Remark 68.13 is valid also for the Reissner-Mindlin system. The remarks concerning 
approximate and exact controllability of the Kirchhoff model utilizing distributed controls remain true 
for the Reissner—-Mindlin system. 


68.6.3 Controllability of the von Karman System 


The global controllability results which hold for the Kirchhoff and Reissner-Mindlin models cannot be 
expected to hold for nonlinear partial differential equations, in general, or for the von Karman system in 
particular. Rather, only local controllability is to be expected (although global controllability results may 
obtain for certain semilinear systems; cf. [19]), that is, in general the most that can be expected is that the 
reachable set (assuming zero initial data) contains some ball S, centered at the origin in the appropriate 
energy space, with the control time depending on r. 

The problem to be considered is 


Ow Ow ‘ 
ph— — I, A + DA2w — [w, G] =0, 
Ot (68.76) 


A’G + > |w,w] = 0, 


—=0, onI,t>0, (68.77) 


of, Ow pay) —™ eee er eae c)+f 
f 12 Ot ee ie ; 


By (68.78) 
iad —v-c, onl),t>0, 
OG 
G=0, —=0 onI,t>0, (68.79) 
ov 
6) 
w(x, 0) = (0) = 0. eR, (68.80) 


It is assumed that 9 4 J. Note that there is no initial data for G. 

The above system is usually analyzed by uncoupling w from G. This is done by solving the second 
equation in Equation 68.79, subject to the boundary conditions, Equation 68.79, for G in terms of w. One 
obtains G = —(Eh/2)G[w, w], where G is an appropriate Green’s operator for the biharmonic equation. 
One then obtains for w the following equation with a cubic nonlinearity: 


Ow Ow Eh 
ph— ~hAae + DA‘w— = Iw, Siw, w]]=0. (68.81) 


The problem for w consists of Equation 68.81 together with the boundary conditions, Equations 68.77, 
68.78, and initial conditions Equation 68.80. 

The function spaces H and V based on the kinetic and strain energy functionals, respectively, related 
to the transverse displacement w, are the same as those introduced in discussing controllability of the 
Kirchhoff model, as is the reachable set R%. corresponding to vanishing data. Let 


S,={(v,h) eV x H: (\lvll} + Ulli? <r}. 


With this notation, a local controllability result analogous to Theorem 68.4 can be established. 
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Theorem 68.6: 


Under assumptions, Equations 68.60, 68.61, and 68.64, there isanr > 0 and a time To(r) > 0 so that 
RS (68.82) 


if T > To. 


Curiously, a result for the von Karman system analogous to Theorem 68.3 is not known. 

Theorem 68.6 is proved by utilizing the global controllability of the linearized (i.e., Kirchhoff) prob- 
lem, together with the implicit function theorem in a manner familiar in the control theory of finite- 
dimensional nonlinear systems. 


Remark 68.19 


If the underlying dynamics are modified by introducing a dissipative term b(x)w, b(x) > 0, into Equa- 
tion 68.81, it may be proved, under assumptions, Equations 68.60 and 68.61, that the conclusion, Equa- 
tion 68.82, is valid for every r > 0. However, the optimal control time To will continue to depend on r, so 
that such a result is still local. 


68.7 Stabilizability of Dynamic Plates 


The problem of stabilization is concerned with the description of feedback controls which assure that 
the trajectories of the system converge asymptotically to an equilibrium state of the system. For infinite- 
dimensional systems in general and distributed parameter systems in particular, there are various distinct 
notions of asymptotic stability: weak, strong, and uniform (distinctions which, incidentally, disappear 
in finite-dimensional approximations of the system). The differences in the various types of stability are 
related to the topology in which convergence to an equilibrium takes place. The most robust notion 
of stability is that of uniform stability, which guarantees that all possible trajectories starting near an 
equilibrium of the system converge to that equilibrium at a uniform rate. In this concept, convergence is 
usually measured in the energy norm associated with the system. This is the classical viewpoint of stability. 
Strong stability, on the other hand, guarantees asymptotic convergence of each trajectory (in the energy 
norm) but at a rate which may become arbitrarily small, depending on the initial state of the system. The 
concept of weak stability is similar; however, in this case, convergence to an equilibrium takes place in 
a topology weaker than associated with the energy norm. In the discussion which ensues, only uniform 
and strong asymptotic stability will be considered. 


68.7.1 Stabilizability of Kirchhoff Plates 


Consider the Kirchoff system consisting of Equation 68.49, boundary conditions, Equations 68.50, and 
68.51, and initial conditions 


w(x, 0) = w°(x), ans 0)=w(x) xeEQ. (68.83) 


It is assumed that '; 4 J, i = 0, 1. The boundary inputs c, f are the controls. The boundary outputs are 


Ow Vv (=) 
y = — 3 ZL — 
Ot T', x(0,00) ot 


The problem is to determine the boundary inputs in terms of the boundary outputs to guarantee that the 
resulting closed-loop system is asymptotically stable in some sense. 


. (68.84) 
T') x (0,00) 
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The total energy of the system at time t is 


E(t) =K(t)+U(t), 


2 
1 2 2 1h 
oe 5 [ on + 1,|Vw| 40+ 535 DL fou Yay Om a 


where w = Ow/0t. A direct calculation shows that 


d€ Ow (6) 
aah. [ Bales +(Fe-o Hf) | e 


When the loop is closed by introducing the proportional feedback law 


f = —koy, c=kz, k;>0, ko tk, >0, (68.85) 


dé aw? aw \? 
i ky | — ko(w)? +k, ( — ) | af, 
dt ial 1) aaa (FR) | 


= =| [ko(w)? +k, |Vwl?] dP <0. 
Ty 


it follows that 


Thus the feedback laws Equation 68.85 are dissipative with respect to the total energy functional €. 

Let H and V be the Hilbert spaces based on the energy functionals K and U, respectively, as introduced 
above. If (w°, w!) € V x H, the Kirchoff system, Equations 68.49 through 68.51, with initial conditions 
Equation 68.83, boundary outputs Equation 68.84, and feedback law Equation 68.85, is well-posed: it has 
a unique finite energy solution w. The system is called uniformly asymptotically stable if there is a positive, 
real-valued function a(t) with a(t) > 0 as t + ov, so that 


(w(t), WE) Iver < a(t) |(w?, w)) Il vce. 


Therefore, E(t) < a7(t)€(0). If such a function a exists, it is necessarily exponential: a(t) = Ce~® for 
some w > 0. The system is strongly asymptotically stable if, for every initial state (w°, w!) € V x H, the 
corresponding solution satisfies 


Jim ow, 7) lv = 0 


or, equivalently, that E(t) + 0 as t > oo. Uniform and strong asymptotic stability are not equivalent 
concepts. Strong asymptotic stability does not imply uniform asymptotic stability. Strong stability has the 
following result. 


Theorem 68.7: 


Assume that k; > 0, i= 0,1. Then the closed-loop Kirchhoff system is strongly asymptotically stable. 


The proof of this theorem amounts to verifying that, under the stated hypotheses, the problem has 
no spectrum on the imaginary axis. The latter is a consequence of the Holmgren uniqueness theorem 
(see [13, Chapter 4] for details). 

For the closed-loop Kirchhoff system to be uniformly asymptotically stable, the geometry of I’; must 
be suitably restricted. 
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Theorem 68.8: 


Assume that k; > 0, i= 0,1, and that Y; satisfy Equations 68.60 and 68.61. Then the closed-loop Kirchhoff 
system is uniformly asymptotically stable. 


The proof of Theorem 68.8 follows from the estimate, 
T 
€(T) < Cr / / [ko(w)? +k |Vw[?]dPdt,  T large. (68.86) 
0 T 


The proof of Equation 68.86 is highly nontrivial. From Equation 68.86 and the above calculation of dé /dt, 


it follows that ‘ 


1+ Cr 


E(T) < E(0), 


which implies the conclusion of the theorem. 


Remark 68.20 


Theorem 68.8 was first proved in [13] under the additional assumption Equation 68.64, but the latter 
condition may be removed by applying the results in [20]. Assumption, Equation 68.61, may be weakened; 
see Remark 68.13. If I, = 0, the conclusion holds even when k; = 0. 


Remark 68.21 


In place of the linear relationship Equation 68.85, one may consider a nonlinear feedback law 
y=-yw), 2z=2(Vw), 
where y(-) is a real-valued function, z(-) : IR? + R? and satisfies 


xy(x) >0, Vx € R\{0}, 
x-z(x) >0 Vx €R?\{O}. 


The closed-loop system is then dissipative. In addition, suppose that both y(-) and z(-) are continuous, 
monotone increasing functions. The closed-loop system is then well-posed in finite energy space. Under 
some additional assumptions on the growth of y(-),z(-) at 0 and at oo, the closed-loop system have a 
decay rate which, however, will be algebraic, rather than exponential, and will depend on a bound on the 
initial data; cf. [13, Chapter 5] and [16]. 


68.7.2 Stabilizability of the Reissner—Mindlin System 
The system, (Equations 68.67 through 68.69) is considered, along with the initial conditions 
O 
w(x, 0) = w(x), S°(x,0) = Ww!) 
es (68.87) 
u(x, 0) = w(x), 5p 9) =ul(x), xeEQ. 
The boundary inputs c, f are the controls. The boundary outputs are 


_ Ow 


_ ou 


zZ= — ‘ 
Ot T1 x (0,00) 


Ot T1x (0,00) , 
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The total energy of the system at time ¢ is 


1 i Lie e 
E(t) =K(t)+U(t) = = ui (phw? + Ip|ul*) dQ + = i: — J° og(u)ej(u) + Ghlu+ Vw)? | dQ. 

ote 2 Jo \ 12 A 

Then i 
E 
AE oak ? wal, 
which suggests that the loop be closed by introducing the proportional feedback law 
f=—koy, c=—kiz, k>0, ko tk, >0, 


so that the closed-loop system is dissipative. In fact, it may be proved that the conclusions of Theorems 
68.7 and 68.8 above hold for this system; see [13], where this is proved under the additional geometric 
assumption, Equation 68.64. 


68.7.3 Stabilizability of the von Karman System 


Consider the system consisting of Equation 68.81, boundary conditions Equations 68.77 and 68.78, and 
initial conditions Equation 68.83. The inputs, outputs, and total energy of this system are defined as for 
the Kirchhoff system, and the loop is closed using the proportional feedback law Equation 68.85. The 
following result has been proved in [4]. 


Theorem 68.9: 


Assume that k; > 0, i= 0,1, and that T; satisfy Equations 68.60, 68.61, and 68.64. Then there is an r > 0 
so that 
E(t) < Ce E(0) (68.88) 


provided E(0) < r, where w > 0 does not depend on r. 


Remark 68.22 


If Ip =0 and To = Y, the estimate Equation 68.88 was established in [13, Chapter 5] for every r > 0, with 
constants C, w independent of r, but under a modified feedback law for c. 


Remark 68.23 


If the underlying dynamics are modified by introducing a dissipative term b(x)w, b(x) > 0, into Equa- 
tion 68.81, it is proven in [5] that, under assumptions, Equations 68.60 and 68.61, the conclusion Equa- 
tion 68.88 is valid for every r > 0, where both constants C, w depend on r. This result was later extended 
in [8] to the case of nonlinear feedback laws (cf: Remark 68.21). 


References 


1. Adams, R.A., Sobolev Spaces, Academic, New York, 1975. 

2. Balakrishnan, A.V., Applied Functional Analysis, 2nd ed., Springer, New York, 1981. 

3. Banks, H.T. and Smith, R.C., Models for control in smart material structures, in Identification and 
Control in Systems Governed by Partial Differential Equations, Banks, H.T., Fabiano, R.H., and Ito K., 
Eds., SIAM, 1992, pp 27-44. 


Controllability of Thin Elastic Beams and Plates 68-31 


15; 


16. 


U7. 


18. 


19. 


20. 


21. 


22. 


23. 


24. 


25. 


26. 
27. 


28. 


Bradley, M. and Lasiecka, I., Local exponential stabilization for a nonlinearly perturbed von Karman 
plate, Nonlinear Analysis: Theory, Methods and Appl., 18, 333-343, 1992. 

Bradley, M. and Lasiecka, I., Global decay rates for the solutions to a von Karman plate without geometric 
constraints, 2 Math. Anal. Appl., 181, 254-276, 1994. 

Bresse, J. A. C., Cours de mechanique applique, Mallet Bachellier, 1859. 

Hirschhorn, M. and Reiss, E., Dynamic buckling of a nonlinear Timoshenko beam, SIAM J. Appl.Math., 
37, 290-305, 1979. 

Horn, M. A. and Lasiecka, I, Nonlinear boundary stabilization of a von Karman plate equation, 
Differential Equations, Dynamical Systems and Control Science: A Festschrift in Honor of Lawrence 
Markus, Elworthy, K. D., Everit, W. N., and Lee, E. B., Eds., Marcel Dekker, New York, 1993, pp 
581-604. 

Kane, T.R., Ryan, R.R., and Barnerjee, A.K., Dynamics of a beam attached to a moving base, AIAA J. 
Guidance, Control Dyn., 10, 139-151, 1987. 

Komornik, V., A new method of exact controllability in short time and applications, Ann. Fac. Sci. 
Toulouse, 10, 415-464, 1989. 

Komornik, V., Exact controllability and stabilization, in The Multiplier Method, Masson - John Wiley & 
Sons, Paris, 1994. 

Krabs, W., On Moment Theory and Controllability of One-Dimensional Vibrating Systems and Heating 
Processes, in Lecture Notes in Control and Information Sciences, Vol. 173, Springer, New York, 1992. 
Lagnese, J. E., Boundary Stabilization of Thin Plates, Studies in Applied Mathematics, SIAM, Philadel- 
phia, 1989, Vol. 10. 

Lagnese, J. E., The Hilbert uniqueness method: A retrospective, in Optimal Control of Partial Differential 
Equations, Hoffmann, K. H. and Krabs, W., Eds., Springer, Berlin, 1991, pp 158-181. 

Lagnese, J. E., Recent progress in exact boundary controllability and uniform stabilizability of thin 
beams and plates, in Distributed Parameter Systems: New Trends and Applications Chen, G., Lee, E. B., 
Littmann, W., and Markus, L., Eds., Marcel Dekker, New York, 1991, pp 61-112. 

Lagnese, J. E. and Leugering, G., Uniform stabilization of a nonlinear beam by nonlinear boundary 
feedback, ] Diff Eqns , 91, 355-388, 1991. 

Lagnese, J. E., Leugering, G., and Schmidt, E.J.P.G., Modeling, Analysis and Control of Dynamic Elastic 
Multi-Link Structures, Birkhauser, Boston-Basel-Berlin, 1994. 

Lagnese, J. E. and Lions, J. L., Modelling, Analysis and Control of Thin Plates, Collection RMA, Masson, 
Paris, 1988, Vol. 6. 

Lasiecka, I. and Triggiani, R., Exact controllability of semilinear abstract systems with application to 
waves and plates boundary control problems, Appl. Math, Opt., 23, 109-154, 1991. 

Lasiecka, I. and Triggiani, R., Sharp trace estimates of solutions to Kirchhoff and Euler-Bernoulli 
equations, Appl. Math, Opt., 28, 277-306, 1993. 

LeDret, H., Problémes variationnels dans les multi-domains: Modelisation des jonctions et applications, 
Collection RMA, Masson, Paris, 1991, vol. 19. 

Lions, J. L., Contrélabilité exacte, perturbations et stabilisation de systémes distribués, Tome I: Contréla- 
bilté exacte, Collection RMA, Masson, Paris, 1988, Vol. 8. 

Lions, J. L., Exact controllability, stabilization and perturbations for distributed systems, SIAM Review, 
30, 1-68, 1988. 

Russell, D. L., On mathematical models for the elastic beam with frequency-proportional damping, in 
Control and Estimation in Distributed Parameter Systems, Banks, H.T., Ed., SIAM, 1992, pp 125-169. 
Triggiani, R., Lack of exact controllability for wave and plate equations with finitely many boundary 
controls, Diff. Int. Equations, 4, 683-705, 1991. 

Washizu, K., Variational Methods in Elasticity and Plasticity, 3rd ed., Pergamon, Oxford, 1982. 
Wempner, G., Mechanics of Solids with Applications to Thin Bodies, Sijthoff and Noordhoff, Rockville, 
MD, 1981. 

Zuazua, E., Contrdlabilité exacte d’un modeéle de plaques vibrantes en un temps arbitrairement petit, C. 
R. Acad. Sci., 304, 173-176, 1987. 


69 


Control of the Heat 
Equation 


RL: TOCA cacasictie ce csseieriexiscestoaieoiedeetensrreecceess 69-1 
69.2 Background: Physical Derivation ..........0..04 69-2 
69.3 Background: Significant Properties ................ 69-5 


The Maximum Principle and Conservation « 
Smoothing and Localization + Linearity « 
Autonomy, Similarity, and Scalings 
69.4 Some Control-Theoretic Examples ............... 69-11 
A Simple Heat Transfer Problem « Exact Control « 
System Identification 
69.5 More Advanced System Theory..........csc0 69-14 
The Duality of Observability/Controllability » The 
One-Dimensional Case » Higher-Dimensional 
Thomas I. Seidman Geometries 
University of Maryland, Baltimore County ETE ROIGE cossiwaraiww itor asia arenciienaehansbalee tants 69-19 


69.1 Introduction 


We must begin by making an important distinction between the considerations appropriate to a great 
variety of practical problems, for example, controlled heat transfer or observed diffusion and those 
more theoretical considerations which arise when we wish to apply the essential ideas developed for the 
control theory of ordinary differential equations in the context of systems governed by partial differential 
equations—here, the linear heat equation 


dv vy ay Oy 3 
oe be Bf” ae fort >0, x=(x,y,z)€EQCR?’. (69.1) 


Many of the former set of problems are ones of optimal design, rather than of dynamic control and 
many of the essential concerns are related to fluid flow in a heat exchanger or to phase changes (e.g., 
condensation) or to other issues which go well beyond the physical situations described by Equation 69.1. 
Some properties of Equation 69.1 are relevant for these problems and we shall mention these, but the 
essential concerns which dominate them are outside the scope of this chapter. 

The primary focus of this chapter is, from the point of view of control theory, on the inherent distinc- 
tions one must make between “lumped parameter systems” (with finite-dimensional state space, governed 
by ordinary differential equations) and “distributed parameter systems” governed by partial differential 
equations such as Equation 69.1 so the state, for each t, is a function of position in the spatial region Q. 
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While Equation 69.1 may be viewed abstractly as an ordinary differential equation* 


=Av+w fort>0, (69.3) 
it is important to realize that abstract ordinary differential equations such as Equation 69.3 are quite 
different in nature from the more familiar ordinary differential equations with finite-dimensional state; 
hence, one’s intuition must be attuned to this situation. Further, the intuition appropriate to consideration 
of the parabolic partial differential equation 69.3 is quite different from what would be appropriate, say, 
for the wave equation 

dw 

dt? 

which describes a very different set of physical phenomena with very different properties (although in 

Section 69.5.3 we do mention an interesting relation for the corresponding theories of observation and 
control). 

We restrict our attention largely to autonomous linear problems for which frequency-domain 


=Aw+wW. fort>0, (69.4) 


approaches involving matrix theory (transfer functions, etc.) and Riccati equations are well-known 
approaches in lumped parameter (finite-dimensional) settings. While we do note that, with appropri- 
ate technical conditions, these approaches generalize to infinite-dimensional settings (note the emphasis 
on operator transfer functions in [3] or on the Riccati equation in [1,6]), we will not consider those 
approaches here, but will concentrate on approaches which are distinctly relevant to considerations of 
partial differential equations, following the approaches of [4,5,7-10], for example. 

One new consideration is that the geometry of the region Q is relevant here. We here concentrate 
primarily on linear problems in which input/output interaction (for control and for observation) is 
restricted to the boundary—partly because this is physically reasonable and partly because it is only for a 
system governed by a partial differential equation that one could even consider “control via the boundary 
conditions.” 

The first two sections of this chapter provide, as background, some relevant properties of Equation 69.1, 
including the presentation of some examples and implications of these general properties for practical 
heat conduction problems. We then turn to the discussion of system-theoretic properties of Equation 69.1 
or 69.3; note that this restricts our discussion to linear problems. We emphasize, in particular, the 
considerations which arise when the input/output occurs in a way which has no direct analog in the theory 
of lumped parameter systems—not through the equation itself, but through the boundary conditions 
which are appropriate to the partial differential equation 69.1. This mode of interaction is, of course, 
quite plausible for physical implementation since it is typically difficult to influence or to observe directly 
the behavior of the system in the interior of a solid spatial region. 


69.2 Background: Physical Derivation 


Unlike situations involving ordinary differential equations with finite-dimensional state space, it is almost 
impossible to work with partial differential equations without developing a deep appreciation for the char- 
acteristic properties of the particular kind of equation. For the classical equations, such as Equation 69.1, 
this is closely related to physical interpretations. Thus, we begin with a discussion of some interpretations 
of Equation 69.1 and only then note the salient properties which will be needed to understand its control. 


* Now v(t) denotes the state, viewed as an element of an infinite-dimensional space of functions on 2, and A = V2 is the 


Laplace operator, given in the 3-dimensional case by 
ay ; ay ay 
Ox2 " Ay2 © Az2 


together with specification of the relevant boundary conditions. 


A:ive (69.2) 
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While we speak of Equation 69.1 as the heat equation, governing conductive heat transfer, our intuition 
will be aided by noting also that this same equation also governs molecular diffusion for dilute solutions 
and certain dispersion phenomena as well as the evolution of the probability distribution in the stochastic 
theory of Brownian motion. 

For heat conduction, we begin with the fundamental notions of heat content Q and temperature, 
related* by 


[heat content] = [heat capacity] - [temperature]. (69.6) 


or, in symbols, 
Q=opcT (69.7) 


where Q is here the heat density (per unit volume), p is the mass density, T is the temperature, and c is the 
“incremental heat capacity” [amount of heat needed to raise the temperature of a unit mass by, say, 1°]. 
The well-known physics of the situation is that heat will flow by conduction from one body to another at a 
rate proportional to the difference of their temperatures. Within a continuum one has a heat flux vector q 
describing the heat flow: q-ndA is the rate (per unit time) at which heat flows through any (imaginary) 
surface element dA, oriented by its unit normal 7. This is now given by Fourier’s Law: 


q = —k grad T = —kVT (69.8) 


with a (constant") coefficient of heat conduction k > 0. 

For any (imaginary) region B in the material, the total rate of heat flow out of B is then f,q-ndA 
(where n is the outward normal to the bounding surface 0B) and, by the Divergence Theorem, this equals 
the volume integral of div gq. Combining this with Equations 69.7 and 69.8—and using the arbitrariness 
of B—this gives the governing* heat equation 


pc— =V-kVT+ (69.9) 


where wf is a possible source term for heat. 

Let us now derive the equation governing molecular diffusion, we consider the spread of some substance 
in another (e.g., a “solute” in a “solvent”) caused, as discussed in one of Einstein’s famous papers of 1905, 
by the random collisions of molecules. Assuming a dilute enough solution that one can neglect the volume 
fraction occupied by the solute in comparison with the solvent, we present our analysis simply in terms of 
the concentration (relative density) C of the relevant chemical component. One has, entirely analogous 


* More precisely, since the mass density p and the incremental heat capacity c (i.e., the amount of heat needed to raise the 


temperature of a unit mass of material by, e.g., 1°C when it is already at temperature 1) are each temperature dependent, 
the heat content in a region R with temperature distribution T(-) is given by 


T 
Q= Q(B) = i i [ocl(¥) d0 dV. (69.5) 


For our present purposes we are assuming that (except, perhaps, for the juxtaposition of regions with dissimilar materials) 
we may take pc to be effectively constant. Essentially, this means that we assume, the temperature variation is not so large 
as to force us to work with the more complicated nonlinear model implied by Equation 69.5. In particular, it means that 
we will not treat situations involving phase changes such as vaporization, condensation, or melting. 
+ This coefficient k is, in general, also temperature dependent as well as a material property. Our earlier assumption in 
connection with pc is relevant here also to permit us to take k to be a constant. 
It is essential to realize that q, as given in Equation 69.8, refers to heat flow relative to the material. If there is spatial motion 
of the material itself, then this argument remains valid provided the regions 6 are taken as moving correspondingly—that 
is, Equation 69.3 holds in material coordinates. When this is referred to stationary coordinates, we view the heat as 
transported in space by the material motion—that is, we have advection as well as diffusion; the Peclet number indicates 
the relative importance of these transport mechanisms. 
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to the previous derivation, a material flux vector J which is now given by Fick’s Law: 
J=—-DVC (69.10) 


where D > 0 is the diffusion coefficient*. As in deriving Equation 69.9, this law for the flux immediately 
leads to the conservation equation 


OG Sos 
- =V-DVC+W%, (69.11) 


where wy is now a source term for this component—say, by some chemical reaction. 

A rather different mechanism for the spread of some substance in another depends on the effect of 
comparatively small relative velocity fluctuations of the medium—for example, gusting in the atmospheric 
spread of the plume from a smokestack or the effect of path variation through the interstices of packed 
soil in considering the spread of a pollutant in groundwater flow. Here one again has a material flux for 
the concentration—given now by Darcy’s Law, which appears identical to Equation 69.10. The situation 
can be more complicated here, however, since one may well have anisotropy (D is then a matrix) and/or 
various forms of degeneracy (e.g., D becoming 0 when C = 0); nevertheless, we still obtain Equation 69.11 
with this dispersion coefficient D. Here, as earlier, we focus on settings where we may take a constant scalar 
D > 0 and neglect advection. 

As we are assuming constant coefficients in each case, we may simplify Equation 69.9 or 69.11 by 
writing these as 

y= DAv+ (69.12) 


where v stands either for the temperature T or the concentration C, subscript ¢ denotes a partial derivative, 
and, in considering Equation 69.9, D stands for the thermal diffusivity a = k/pc. We may, of course, always 
choose units to make D = 1 in Equation 69.12; hence, it becomes precisely Equation 69.3. It is interesting 
and important for applications to have some idea of the wide range of magnitudes of the coefficient D in 
Equation 69.12 in fixed units—say, cm? /s.—for various situations. For heat conduction, typical values of 
the coefficient D = a are, quite approximately: 


8.4 for heat conduction in diamond; 1.1 for copper; 0.2-0.6 for steam (rising with temper- 
ature); 0.17 for cast iron and 0.086 for bronze; 0.011 for ice; 7.8 x 10~ for glass; 4 x 10-3 
for soil; 1.4-1.7 x 1073 for water; 6.2 x 10~4 for hard rubber; and so on. 


For molecular diffusion, typical figures for D might be 


around 0.1 for many cases of gaseous diffusion; 0.28 for the diffusion of water vapor in air 
and 2 x 10~> for air dissolved in water; 2 x 10~° for a dilute solution of water in ethanol 
and 8.4 x 10~° for ethanol in water; 1.5 x 1078 for solid diffusion of carbon in iron and 
1.6 x 10~!° for hydrogen in glass, and so on. 


Finally, for example, the dispersion of a smoke plume in mildly stable atmosphere (say, a 15 mph breeze) 
might, on the other hand, have D approximately 10° cm?/s.—as might be expected, dispersion is a far 
more effective spreading mechanism than molecular diffusion, but the same mathematical description 
covers both. 

Assuming that one knows the initial state of the physical system 


v(x,t=0)=vo(x) onQ, (69.13) 


where Q is the region of R? we wish to consider, we still cannot expect to determine the system evolution 
unless we also know (or can determine) the source term and, unless Q would be all of R°, can furnish 


* More detailed treatments might consider the possibility that D depends on the temperature, and so on, of the solvent and 


is quite possibly also dependent on the existing concentration, even for dilute concentrations. As earlier, we neglect these 
effects as insignificant for the situations under consideration and take D to be constant. 
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adequate information about the interaction at the boundary OQ. The simplest setting is that there is to 
be no such interaction at all: the physical system is to be insulated from the rest of the universe and thus 
there is no flux across the boundary. Formally, this means q-n = 0, where 7 is now the unit normal to 
OQ or, from Equation 69.11 or 69.8 with the scaling of Equation 69.12, 


D— =—DVv-n=0. (69.14) 


More generally, the flux might be more arbitrary but known; hence we have the inhomogeneous Neumann 
condition: 


—-D—=g, onY=(0,T) x dQ. (69.15) 


An alternative* set of data would involve knowing the temperature (concentration) at the boundary, that 
is, having the Dirichlet condition: 


v=go on X=(0,T) x OQ. (69.17) 


The mathematical theory supports our physical interpretation: 


If we have Equation 69.1 on Q = (0, T) x Q with \ specified on Q and the initial condition 
Equation 69.13 specified on Q, then either* of the boundary conditions Equation 69.15 or 
69.17 suffices to determine the evolution of the system on Q, that is, for0 < t < T. 


We refer to either of these as the direct problem. An important property of this problem is that it is 
well-posed, that is, a unique solution exists for each choice of the data and small changes in the data 
produce * correspondingly small changes in the solution. 


69.3 Background: Significant Properties 


In this section we note some of the characteristic properties of the “direct problem” for the partial 
differential equation 69.1 and, related to these, introduce the representation formulas underlying the 
mathematical treatment. 


69.3.1 The Maximum Principle and Conservation 


One characteristic property, going back to the physical derivation, is that Equation 69.1 is a conservation 
equation. In the simplest form, when heat or material is neither created nor destroyed in the interior 


* Slightly more plausible, physically, would be to assume that the ambient temperature or concentration would be known 


or determinable to be g “just outside” OQ and then to use the flux law (proportionality to the difference) directly: 
Ov 


Da, =q:n=Mv—g) ond (69.16) 


with a flux transfer coefficient .. > 0. Note that, if ) ~ 0 (negligible heat or material transport), then we effectively get 
Equation 69.14. On the other hand, if \ is very large (v— g = —(D/2)Ov/On with D/d © 0), then v will immediately 
tend to match g at OQ, giving Equation 69.17; see Section 69.4.1 

We may also have, more generally, a partition of OQ into [9 UT, with data given in the form Equation 69.17 on 
Xo = (0, T) x Po and in the form Equation 69.15 on 2; = (0,T) x Ty. 

We note that making this precise—that is, specifying the appropriate meanings of “small”—becomes rather technical and, 
unlike the situation for ordinary differential equations, can be done in several ways which each may be useful for different 
situations. We will see, on the other hand, that some other problems which arise in system-theoretic analysis turn out to 
be “ill-posed”, that is, not to have this well-posedness property; see, for example, Section 69.4.3. 
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( = 0) and if the region is insulated Equation 69.14, then [total heat or material] = {, v dV is constant 


in time. More generally, we have 
d 
— [vav =i aida [ wdV (69.18) 
dt LJg an Q 


for v satisfying Equations 69.3 through 69.15. 
Another important property is the Maximum Principle: 


Let v satisfy Equation 69.3 with y > 0 on Q, :=(0,t) x Q. Then the minimum value of 
v(t,x) on QO, is attained either initially (t = 0) or at the boundary (x € OQ). Unless v is a 
constant, this value cannot also occur in the interior of Q,; if it is a boundary minimum with 
t > 0, then one must have Ov/On > 0 at that point. Similarly, if v satisfies Equation 69.3 with 
W <0, then its maximum is attained for t = 0 or at x € OQ, and so on. 


One simple argument for this rests on the observation that at an interior minimum one would neces- 
sarily have 1, = Ov/Ot < 0 and also Av > 0. 

The Maximum Principle shows, for example, that the mathematics of Equation 69.1 is consistent with 
the requirement for physical interpretation that a concentration cannot become negative and the fact 
that, since heat flows “from hotter to cooler,” it is impossible to develop a “hot spot” except by providing 
a heat source. 


69.3.2 Smoothing and Localization 


Perhaps the dominant feature of Equation 69.1 is that solutions rapidly smooth out, with peaks and valleys 
of the initial data flattening out. We will see this in more mathematical detail later, but comment now on 
three points: 


« Approach to steady state 
¢ Infinite propagation speed 
¢ Localization and geometric reduction 


The first simply means that if neither nor the data gy would vary in time, then the solution v of 
Equations 69.3 through 69.17 on (0,00) x Q would tend, as tf + oo, to the unique solution v of the 
(elliptic) steady-state equation 


alg laa i (69.19) 
— 5: Vv = y o 
Ox* — Oy* — Oz? aq © 


The timescale of this transient is given by the lowest eigenvalue of the Laplace operator here. Essentially 
the same would hold if we were to use Equation 69.15 rather than Equation 69.17 except that, as is obvious 
from Equation 69.18, we must then impose a consistency condition that 


: sida [ wdV=0 
OQ Q 


for there to be a steady state at all—and then must note that the solution of the steady-state equation 


E rv PD 


ax * By * | Fe ee oe 


only becomes unique when one supplements Equation 69.20 by specifying, from the initial conditions 
(Equation 69.13), the value of [, vdV. 

Unlike the situation with the wave equation 69.4, the mathematical formulation (Equation 69.1), and 
so on, implies an infinite propagation speed for disturbances—for example, the effect of a change in the 
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boundary data go(t, x) at some point x, € OQ occurring at a time t = f, is immediately felt throughout 
the region, affecting the solution for every x € 2 at every t > t,. One can see that this is necessary to have 
the Maximum Principle, for example, but it is certainly nonphysical. This phenomenon is a consequence 
of idealizations in our derivation and becomes consistent with our physical intuition when we note that 
this “immediate influence” is extremely small: there is, indeed, a noticeable delay before a perturbation 
will have a noticeable effect at a distance. 

Consistent with the last observation, we note that the behavior in any subregion will, to a great extent, 
be affected only very slightly (in any fixed time) by what happens at parts of the boundary which may 
be very far away; this is a sort of “localization” principle. For example, if we are only interested in what 
is happening close to one part of the boundary, then we may effectively treat the far boundary as “at 
infinity.” To the extent that there is little spatial variation in the data at the nearby part of the boundary, 
we may then approximate the solution quite well by looking at the solution of the problem considered 
on a half-space with spatially constant boundary data, dependent only on time. Taking coordinates so 
that the boundary becomes the plane “x = 0,” one easily sees that this solution will be independent of 
the variables y, z if the initial data and source term are. Equation 69.1 then reduces to a one-dimensional 
form 

Ov Oy 

Ot Ax? 
for t > 0 and, now, x > 0 with, for example, specification of v(t, 0) = go(t) and of v(0, x) = vo(x). Similar 
dimensional reductions occur in other contexts—one might obtain Equation 69.21 for 0 < x < L, where L 
gives the thickness of a slab in appropriate units or one might get a two-dimensional form corresponding 
to a body which is long compared to its constant constant cross-section and with data which is relatively 
constant longitudinally. In any case, our equation will be Equation 69.3, with the dimensionally suitable 
interpretation of the Laplace operator. Even if the initial data does depend on the variables to be omit- 
ted, our first property asserts that this variation will tend to disappear; hence we may still get a good 
approximation after waiting through an initial transient. On the other hand, one usually cannot accept 
this approximation near, for example, the ends of the body where “end effects” due to those boundary 
conditions may become significant. 


+ (t,x) (69.21) 


69.3.3 Linearity 


We follow Fourier in using the linearity of the heat equation, expressed as a “superposition principle” for 
solutions, to obtain a general representation for solutions as an infinite series. Let {[e,, 4g]: k =0,1,...} 
be the pairs of eigenfunctions and eigenvalues for —A on &, that is, 


—Ae, = rper on Q (with BC) fork =0,1,... (69.22) 
where “BC” denotes one of the homogeneous conditions 
Oe 
ee =0 or —*=0 onda (69.23) 
on 


according as we are considering Equation 69.17 or 69.15. It is always possible to take these so that 
i: le. |? dV = 1, i ee, dV =0 foriék, (69.24) 
Q Q 


with 0 < do < Ay <---> 00 we have Ag > 0 for Equation 69.17 and io = 0 for Equation 69.15. 
One sees immediately from Equation 69.22 that each function e~**"e;(x) satisfies Equation 69.1; hence 
superposing, we see that 


v(t,x) = s che ex (x) (69.25) 
k 
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gives the “general solution” with the coefficients (c,) obtained from Equation 69.13 by 


ck = (ek ¥0) 80 vo(-) = D> cee), (69.26) 


assuming Equation 69.24. Note that (-, -) denotes the L?(Q) inner product: (f,g) = Jof g(x) dx (for 
m-dimensional {2—with, physically, m = 1,2, 3). The expansion (Equation 69.26, and so Equation 69.25), 
is valid if the function vp is in the Hilbert space L?(Q), that is, if te |vo|? < co. Note that the series 
(Equation 69.26) need not converge pointwise unless one assumes more smoothness for vp but, since it is 
known that, asymptotically as k + oo, one has 


he~ CK/™ with C = C(Q), (69.27) 


the factors e~**' decrease quite rapidly for any fixed t > 0 and Equation 69.25 then converges nicely to a 
smooth function. Indeed, this is just the “smoothing” noted above: this argument can be used to show that 
solutions of Equation 69.1 are analytic (representable locally by convergent power series) in the interior 
of Q for any t > 0 and we note that this does not depend on having homogeneous boundary conditions. 

Essentially the same approach can be used when there is a source term f as in Equation 69.9 but we still 
have homogeneous boundary conditions as, for example, go = 0 in Equation 69.17. We can then obtain 
the more general representation 


v(t,x) = ys yx(t)ex(x), where 


t 
vale) = cet + f eo KE) by (5) ds, (69.28) 
0 


Ck = (€k, Vo), W(t) = (ex, WE, -)) 


for the solution of Equation 69.9. When jf is constant in ft, this reduces to 


yult) = Wie / de + [ck — We/ de] & HE —> WE/ dx 


which not only shows that v(t,-) > v, as in Equation 69.19 with gop = 0, but also demonstrates the 
exponential rate of convergence with the transient dominated by the principal terms, corresponding to 
the smaller eigenvalues. This last must be modified slightly when using Equation 69.15, since one then 
has kp = 0. 

Another consequence of linearity is that the effect of a perturbation is simply additive: if ¥ is the solution 
of Equation 69.9 with data y and % and one perturbs this to obtain a new perturbed solution 7 for the 
data ty + and % + vp (and unperturbed boundary data), then the solution perturbation v = 7 — ¢ itself 
satisfies Equation 69.9 with data and vo and homogeneous boundary conditions. If we now multiply 
the partial differential equation by v and integrate, we obtain 


4 (Loe) fot fw 


using the divergence theorem to see that f vAv = — [ |Vv|2 with no boundary term since the boundary 
conditions are homogeneous. The Cauchy-Schwarz inequality gives | te vy < ||v]] WI], where || - || is the 


L?(Q)-norm: ||v|| = [ de Ta My and we can then apply the Gronwall Inequality* to obtain, for example, 


the energy inequality 
t t 
IIv(e)II7, 2 i IvI? ds < (Iol?+ i Iwi? as) et. (69.29) 
0 0 


This is one form of the well-posedness property asserted at the end of the last section. 


* Ifa function @ > 0 satisfies p(t) < C+M fj ¢(s) ds for 0 < t < T, then it satisfies: p(t) < CeM! there. 
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69.3.4 Autonomy, Similarity, and Scalings 


Two additional useful properties of the heat equation are autonomy and causality. The first just means that 
the equation itself is time independent; hence a time-shifted setting just gives the time-shifted solution. For 
the pure initial-value problem—that is, Equation 69.1 with g = 0 in Equation 69.17 or 69.15—“causality” 
means that v(t, -) is determined by its “initial data” at any previous time fo; hence we may write 


v(t, -) = S(t — to) v(to, -) (69.30) 


where S(t) is the solution operator for Equation 69.1 for elapsed time t > 0. This operator S(t) is a nice 
linear operator in a variety of settings, for example, L?(&) or the space C(&2) of continuous functions with 
convergence of functions meaning uniform convergence. A comparison with Equation 69.25 shows that 


S(t) : eg eM o, so S(t) bs ae — bs ue : (69.31) 
k k 


From Equation 69.30 one obtains the fundamental “semigroup property” 
S(s +t) =S(t)oS(s) fort,s>0. (69.32) 


This only means that, if one initiates Equation 69.1 with any initial data vo at time 0 and so obtains 
v(s,-) = S(s)vo after a time s and v(s+t,-) = S(s + f)vo after a longer time interval of length s + ft, as in 
Equation 69.30, “causality” gives v(s + t,-) = S(t)v(s, -). It is possible to verify that this operator function 
is strongly continuous at t = 0: 


S(Hvo > vo ast—0_ for each vo 


and is differentiable for t > 0: Equation 69.1 just tells us that 
d 
ap) = AS(t), (69.33) 


where the Laplace operator A here includes specification of the appropriate boundary conditions; we 
refer to A in Equation 69.33 as “the infinitesimal generator of the semigroup S(-).” 
In terms of S(-) we obtain a new solution representation for Equation 69.9: 


t 
v(t, -) = S()vo +f S(t — s)W(s, -) ds. (69.34) 
0 


Note that S precisely corresponds to the “Fundamental Solution of the homogeneous equation” for 
ordinary differential equations and Equation 69.35 is just the usual “variation of parameters” solution for 
the inhomogeneous equation 69.9; compare also with Equation 69.33. We may also treat the system with 
inhomogeneous boundary conditions by introducing the Green’s operator G : g +> w, defined by solving 


—Aw=0o0nQ, Bw=gataQ. (69.35) 


with Bw either w or Ow/On, according as we consider Equation 69.17 or 69.15. Since u = v — w then 
satisfies u; = Au+ (vy — w;) with homogeneous boundary conditions, we may use Equations 69.33 and 
69.34 to obtain, after an integration by parts, 


t t 
v(t, -) = S(t)vo + Glgo(t) — go(0)] ds +f S(t — s)W(s, -) ds — / AS(t — s)Ggo(s) ds. (69.36) 
0 0 


The autonomy/causality above corresponds to the invariance of Equation 69.1 under time-shifting and we 
now note the invariance under some other transformations. For this, we temporarily ignore considerations 
related to the domain boundary and take Q to be the whole 3-dimensional space R?. 
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It is immediate that in considering Equation 69.12 (with y= 0) with constant coefficients we have 
ensured that we may shift solutions arbitrarily in space. Not quite as obvious mathematically is the phys- 
ically obvious fact that we may rotate in space. In particular, we may consider solutions which, spatially, 
depend only on the distance from the origin; so v = v(t, r) with r = |x| = \/x? + y? + 2. Equation 69.12 


with | = W(t, 1) is then equivalent to 
dv av 2av 
=D + +0, (69.37) 


Ot Or? r Or 


which involves only a single spatial variable. For the two-dimensional setting x = (x, y) as in Section 69.3.2, 
this becomes 


Oop oie! OU (69.38) 
ot | dr2. r Or us ‘ 


More generally, for a d-dimensional case, Equations 69.37 and 69.38 can be written as 
y= ED (r4"1y,) +. (69.39) 
r 


The apparent singularity of these equations as r — 0 is, of course, only an effect of the use of polar 
coordinates. As in Section 69.3.3, we may seek a series representation like Equation 69.25 for solutions 
of Equation 69.39 with the role of the eigenfunction equation 69.22 now played by Bessel’s equation; we 
then obtain an expansion in Bessel functions with the exponentially decaying time dependence e~*# 
earlier. 

Finally, we may also make a combined scaling of both time and space. If, for some constant c, we set 


as 


t=cDt, %=cx, (69.40) 


then, for any solution v of Equation 69.12 with w= 0, the function (¢,%) = v(t,x) will satisfy 
Equation 69.1 in the new variables. This corresponds to the earlier comment that we may make D= 1 
by appropriate choice of units. 

Closely related to the above is the observation that the function 


k(t,x) = (4nDt)~4/2¢-s°/408 (69.41) 


satisfies Equation 69.39 for t > 0 while a simple computation shows” that 
[ k(t,x)dgx =1 for eacht > 0; (69.42) 
Rd 


thus k(t,-) becomes a 8-function as t > 0. So, k(t—s,x—y) is the impulse response function for an 
impulse at (s, y). Taking d = 3, we note that 


En= a _K(t.x—y)voly) day (69,43) 


is a superposition of solutions (now by integration, rather than by summation); hence, linearity ensures 
that v is itself a solution; we also have 


v(t,-) —> v9 ast— 0, (69.44) 


where the specific interpretation of this convergence depends on how smooth vp is assumed to be. Thus, 
Equation 69.43 provides another solution representation—although, as noted, it ignores the effect of the 
boundary for a physical region which is not all of R°. For practical purposes, following the ideas of 
Section 69.3.2, the formula (Equation 69.43) will be a good approximation to the solution so long as 
./2Dt is quite small* as compared to the distance from the point x to the boundary of the region. 


* We may observe that k(t, -) is a multivariate normal distribution (Gaussian) with standard deviation /2Dt > Oast > 0. 


+ When x is too close to the boundary for this to work well, it is often plausible to think of OQ as “almost flat” on the 
relevant spatial scale and then to extend vg by reflection across it—as an odd function if one were using Equation 69.17 
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69.4 Some Control-Theoretic Examples 


In this section we provide three comparatively elementary examples to see how the considerations above 
apply to some control-theoretic questions. The first relates to a simplified version of a quite practical 
heat transfer problem and is treated with the use of rough approximations, essentially to see how such 
heuristic treatment can be used to obtain practical results. The second describes the problem of control 
to a specified terminal state—which would be a standard problem in the case of ordinary differential 
equations but which involves some new considerations in this distributed parameter setting. The final 
example is a “coefficient identification” problem: using interaction (input/output) at the boundary to 
determine the function q = q(x) in an equation of the form u; = uxx — qu, generalizing Equation 69.21. 


69.4.1 A Simple Heat Transfer Problem 


We consider a slab of thickness a and diffusion coefficient D within which heat is generated at constant 
rate . On the one side, this is insulated (v, = 0) and on the other, it is in contact with a stream of coolant 
(diffusion coefficient D’) moving in an adjacent duct with constant flow rate F in the y-direction. Thus, 
the slab occupies {(x, y): 0 <x <a, 0<y < L}and the duct occupies {(x,y):a<x<a,0<y< L} with 
a,a « Land no dependence on z. 

If the coolant enters the duct at y = 0 with input temperature uo, our problem is to determine how hot 
the slab will become. For this purpose, we assume that we are operating in steady state, that the coolant 
flow is turbulent enough to ensure perfect mixing (and so constant temperature) across the duct, and 
that—to a first approximation—the longitudinal transfer of heat is entirely by the coolant flow so we may 
consider conduction in the slab only in the transverse direction (0 < x < a). 

The source term jf in the slab gives heat production a per unit distance in y and this must be carried 
off by the coolant stream to have a steady state. We might, as noted earlier, shift to material coordinates 
in the stream to obtain an equation there but, more simply, we just observe that when the coolant has 
reached the point y it must have absorbed the amount ay of heat per second and, for a flow rate F 
(choosing units so that pc in Equation 69.7 is 1) this will have raised the coolant temperature from ug to 
[uo + ay /F] =: u(y). 

Now consider the transverse conduction in the slab. We have there v1; = Dv, + W with v; = 0 for 
steady state. As vy = 0 at the outer boundary x = 0, the solution has the form v = v* — (y/2D)x?, where 
v* is exactly what we wish to determine. If we assume a simple temperature match of slab to coolant 
(v = u(y) at x = a), then this gives v*(y) — (/2D)a* = v(a,y) = u(y) = up + alby/F; so 


vt =v") = uo + [2+] aw, 
esa balay (69.46) 
vom [245 (-[] ) aw. 


A slight correction of this derivation is worth noting: for the coolant flow we expect a bound- 
ary layer (say, of thickness 8) of “stagnant” coolant at the duct wall and within this layer we have 
ux © constant = — [v(a) —u low | /8, while also —D'ux = flux= aw by Fourier’s Law; so, instead of 
matching v(a) = u(y), we obtain v(a) = u(y) + (8/D’)a which also increases v*,v by (8/D’/)ay as a 
correction to Equation 69.46; effectively, this correction notes the reduction of heat transfer through 


with go = 0 or as an even function if one were using Equation 69.15 with g; = 0. For Equation 69.17 with, say, go = go(t) 
locally, there would then be a further correction by adding 


t. A 
[ k(t — s,x)go(s) ds, ae, a ee ee a (69.45) 
0 Tt Ox 


where ky = kj(t,x) is as in Equation 69.41 for d=1 and x is here the distance from x to the boundary; compare 
Equation 69.36. There are also comparable correction formulas for more complicated settings. 
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replacing the boundary conditions (Equation 69.17) by (Equation 69.16) with } = D’/8. Much more 
complicated corrections would be needed if one would have to consider conduction within the duct, 
especially with a velocity profile other than the plug flow assumed here. 

We also note that in this derivation we neglected longitudinal conduction in the slab, essentially 
omitting the vy, term in Equation 69.12. Since Equation 69.46 gives v,, = 0, this is consistent with the 
equation. It is, however, inconsistent with reasonable boundary conditions at the ends of the slab (y = 0, L) 
and one would expect “end effects” as well as some evening out of v*. 

We note that, although this was derived in steady state, we could think of using Equation 69.46 for an 
optimal control problem (especially if y would be time dependent, but slowly varying) with the flow rate 
F as control. 


69.4.2 Exact Control 


We consider the problem of using if as control to reach a specified “target state,” w = w(x) at time T. We 
base the discussion on the representation* (Equation 69.28), which permits us to treat each component 
independently: the condition that v(T, -) = w becomes the sequence of “moment equations” 


T 
— xT —hx(T-s) 
y(T) — ce oh i e i Wx(s) ds (69.48) 


= Wk i= (ek, @) 


for each k. This does not determine the control uniquely, when one exists, so we select by optimality, 
minimizing the norm of ty in L?(Q) with Q = (0, T) x Q. This turns out to be equivalent to requiring 
that W,(t) should be a constant times e**?—); so, noting that fe Jes) |? ds = [1 — gone [2s 
the conditions (Equation 69.48) give us the formula 


_ —h_T 
Wk — Che = az 
wt, ae y ( aut )e MTD @ (x), (69.49) 


This formula converges if (and only if) the specified target state w is, in fact, attainable by some control 
in L?(Q). So far, so good! 

Let us now see what happens when we actually attempt to implement the use of Equation 69.49. 
Adding a touch of realism, one must truncate the expansion (say, at k = K) and one must then find each 
coefficient a, = w —c,e~**? with an error bound x by using some algorithm of numerical integration 
on {2 to compute the inner products (e;,). [For simplicity we assume that we would already know 
exactly the relevant eigenvalues and eigenfunctions, as is the case for Equation 69.21 and for a variety of 
higher-dimensional geometries.] Denoting the optimal control by Y and the approximation obtained by 
Wx, we can bound the total error by 


2 = EEE AEE 
MSY | ene | 42 eee | (69.50) 


kal k>K 


For an attainable target w the second sum is small for large K, corresponding to convergence of 
Equation 69.49. The use of a fixed error bound |e,| < ¢ for the coefficient computation would make the 
first sum of the order of K!+@/4¢ by Equation 69.27 so this sum would become large as K increased. 


* For definiteness, one may think of the 1-dimensional heat equation 69.21 with homogeneous Dirichlet boundary condi- 


tions at x = 0, 1. The eigenvalues and normalized eigenfunctions are then 
R= kex?, eg (x) = V2 sin kxx; (69.47) 


so the expansions, starting atk = 1, for convenience, are standard Fourier sine series. 
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To make the total error (Equation 69.50) small requires picking K and then choosing ¢ dependent on 
this choice—or using a relative error condition: |e, | < ¢|a;|. This last seems quite plausible for numerical 
integration with floating point arithmetic—but one trap remains! Neglecting vo, a plausible form of the 
error estimate for a method of numerical integration might be 


2 
lex] < Ch” loex lity) ~ CAA” lleollpo 


where h characterizes a mesh size and the subscript on || - ||[yj indicates consideration of derivatives of 
order up to v, with v depending on the choice of integration method; we have noted that ||ex||[vj ~ at! : 


since the differential operator A is already of order 2. This means that one might have to refine the mesh 
progressively to obtain such a uniform relative error for large k. 


69.4.3 System Identification 


Finally, we consider a 1-dimensional example governed by an equation known to have the form* 


Ov Ov 

a = Dio — q(x)v, (69.51) 
but with D and the specific coefficient function q(-) unknown or known with inadequate accuracy. We 
will assume here that u = 0 at x = 1, but that interaction is possible at the end x = 0 where one can both 
manipulate the temperature and observe the resulting heat flux; for simplicity, we assume that vp = 0. 
Thus, we consider the input/output pairing: g > f, defined through Equation 69.51 with 


u(t,1)=0 u(t, 0) = g(t) 


(69.52) 
f(t) := —Du,(t, 0). 


By linearity, causality, and autonomy of Equation 69.51, we see that this pairing takes the convolution 
form 


t t 
f@H= / o(t —s)g(s) ds = i o(s)g(t — s) ds (69.53) 
0 0 


where o(-) is a kind of impulse response function. Much as we obtained Equations 69.25 and 69.36, we 
obtain 


o(t)= > ope Ee 
k 


with 0, := —Ddxe;,(0)(z, ex) (69.54) 
Dz” —qz=0 2z(0)=1, z(1)=0, 
—Dey + ger = reek ex (0) = 0 = ex(1), 


noting that z and {(A,, e,)} are unknown since q is unknown. 

Viewing Equation 69.53 as an integral equation for o, it can be shown that Equation 69.53 determines o 
for appropriate choices of the input g(-)—simplest would be if we could take g to be a 8-function (impulse) 
so that the observed f would just be o: otherwise we must first solve a Volterra equation of first kind, 
which is already an ill-posed problem. The function o(-) contains all the information about the unknown 
q which we can get and it is possible to show that quite large differences for q may produce only very 
small perturbations of o; thus, this identification problem cannot be “well-posed,” regardless of g(-). 

None of the coefficients 0, will be 0; so, given o(-), Equation 69.54 uniquely determines the eigenvalues 
{4} which appear there as exponents. We note that 4, ~ Dnk?;so D = lim, 4;./17k? is then determined. 


* For example, such an equation might arise for a rod reduced to a simplified 1-dimensional form with heat loss to the 


environment appearing through a boundary condition at the surface of the rod as in Equation 69.16, with g = constant 
and X, here q, varying along the rod. 
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It is then possible to show (by an argument involving analytic continuation, Fourier transforms, and 
properties of the corresponding wave equation) that o(-) uniquely determines q(-), as desired. 

The discussion above gives no suggestion as to how to compute D, q(-) from the observations. Typically, 
one seeks nodal values for a discretization of g. This can be done, for example, by history matching, an 
approach often used for such identification problems, in which one solves the direct problem with a 
guessed q to obtain a resulting “f = f(q)” and proceeds to find the q which makes this best match the 
observed f. It is a useful viewpoint to consider the guessed q as a control so that the idea of ‘best match’ 
makes this an optimal control problem 

With some further a priori information about the function q—say, a known bound on the deriva- 
tive q’—the uniqueness result, although itself nonconstructive, serves to ensure convergence for these 
computations to the correct result as the discretization is refined. Note that it is the auxiliary a priori 
information which converts this to a well-posed problem, although one which will be quite badly condi- 
tioned; hence the practical difficulties do not entirely disappear. A frequently used approach to this is the 
use of regularization, for example, including in the optimization criterion for “best match” some penalty 
for undesirable oscillations of q, trading resolution for stability of the computations. 


69.5 More Advanced System Theory 


In this section we consider the system-theoretic results available for the heat equation, especially regarding 
observability and controllability. Our emphasis is on how, although the relevant questions are quite 
parallel to those standard in “lumped parameter” control theory, one has new technical difficulties which 
can occur only because of the infinite-dimensional state space; this will also mean that this section 
describes in more detail the results of Functional Analysis* and special mathematical results for the 
partial differential equations involved. 


69.5.1 The Duality of Observability/Controllability 


For the finite-dimensional case, controllability for a problem and observability for the adjoint problem 
are dual—essentially, one can control x = Ax + Bg (g(-) =control) from one arbitrary state to another if 
and only if only the trivial solution of the adjoint equation —y = A*y can give [observation] = B*y(-) = 0. 
Something similar holds for the heat equation (e.g., with boundary I/O), but we must be rather careful in 
our statement. 

We begin by computing the relevant adjoint problem, taking the boundary control problem as 


u,= Au on Qwith Bu=gon® and u ay uo (69.55) 
in which the control function g is the data for the boundary conditions, defined on & = (0, T) x OQ. 
As for Equation 69.35, the operator B will correspond to either Equation 69.17 or 69.15; we may now 
further include in the specification of B a requirement that g(-) is restricted to be 0 outside some fixed 
“patch”—that is, a relatively open subset U/ C OQ, viewed as an “accessible” portion of 92—and then 
refer to this as a problem of “boundary patch control.” Note that 


ur := u(T,-) = S(T)uo + Lg(-) (69.56) 


where Equation 69.36 gives 


T 
L: g() > Glg(T) — g(0)] [ AS(T —s)G¢(s) ds. 


* We note [1,2] as possible general references for Functional Analysis, specifically directed toward distributed parameter 


system theory. 
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For the adjoint problem, we consider 
—vy,=Av Bv=0; g= [Bv] ie (69.57) 


where B gives the “complementary” boundary data: Bv := Ov/n if B corresponds to Equation 69.17 and 


By := V1 0 if B corresponds to Equation 69.15. We then have, using the Divergence Theorem, 


| iE urve || ip wor |= folur = fol wv — Hv) 


= fyluav—uva] =— fi go, 


where we write vr, vo for v(T, -), v(0, -), respectively, and set QO = (0, T) x Q. Thus, with subscripts indi- 
cating the domain for the inner product of L7(-), we have the identity 


(ur, VT) 2 + (g, Pu = (Uo, Vo) 2 (69.58) 


from which we wish to draw conclusions. 

First, consider the reachable set R = {ur : g =any € L?(U); up = 0}, which is just the range of the 
operator L: L?(U) > L?(Q). If this were not dense, that is, if we did not have R = L?(Q), then (by the 
Hahn-Banach Theorem) there would be some nonzero v7. orthogonal to all ur € R; so Equation 69.58 
would give (g, ~*)z = 0 for all g, whence ~* = 0, violating detectability (i.e., that e* = 0 only if v* = 0). 
Conversely, a violation of detectability would give a nonzero v* with vj. #0 orthogonal to R. Thus, 
detectability is equivalent to approximate controllability. This last means that one could control arbitrarily 
closely to any target state, even if it cannot be reached exactly—a meaningless distinction for finite- 
dimensional linear systems although significant for the heat equation since, as we have already noted, 
solutions of Equation 69.1 are very smooth (analytic) in the interior of Q; so only very special targets 
could be exactly reachable. 

Detectability means that the map vr +> vr gis 1-1; hence, inverting, p> vr > vo is well-defined: 
one can predict (note the time-reversal in Equation 69.57) vo from observation of ¢ on U. In the finite- 
dimensional case, any linear map such as A: g+> vo would necessarily be continuous (bounded), but 
here this is not automatically the case; note that the natural domain of A is the range of vr + ¢ and, if 
one had continuity, this would extend to the closure M = My C L?(U). For bounded A: M > L?(Q) 
there is a bounded adjoint operator A* : L?(Q) > M and, if we were to set g = A* uo in Equation 69.55, 
we would obtain 


(ur, Vr)Q = (uo, AG)@ — (A*u, )y =0 for every vr € L7(Q). 


This would imply uy = 0; hence g = A* up is a nullcontrol from uo—indeed, it turns out that this g is the 
optimal nullcontrol in the sense of minimizing the L?(/)-norm. Conversely, if there is some nullcontrol 
g for each up, there will be a minimum-norm nullcontrol g and the map C : ug +> g is then linear and is 
continuous by the Closed Graph Theorem. Further, its adjoint A = C* is just the observation operator: 
(+> vr whence bounded observability for the adjoint problem is equivalent to nullcontrollability* for 
Equation 69.55 which, from Equation 69.56, is equivalent to knowing that the range of L contains the 
range of S(T). 

Suppose we have nullcontrollability for arbitrarily small T > 0, always taking U/ = [0, T] x U for some 
fixed patch U C 0Q. A simple argument shows that the reachable set R must then be entirely independent 
of T and of the initial state up. No satisfactory characterization of R is available, although there are various 
known sufficient considerations to have some w € R. 


* This observation is the heart of the Hilbert Uniqueness Method (HUM) introduced by J.-L. Lions. If one has detectability 
one might use the great freedom Hilbert space theory gives for selecting norms to find one making the space of observations 
complete so C* would be bounded; then, if the resultant dual space can be suitably characterized, one will have found an 
appropriate context for nullcontrollability. 
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69.5.2 The One-Dimensional Case 


From the discussion above, it will clearly be sufficient to prove bounded observability for the one- 
dimensional heat equation to have nullcontrollability also. (As a historical note, this equivalence was not 
realized at the time these results were first proved; so originally they were proved independently.) We will 
consider specifically the observability problem with Q = (0, 1), U = (0, T) x {0} and 


Vp=Vxx V(t,0)=v(t,1)=0; g(t):=v,(t, 0), (69.59) 


for which we explicitly know Equation 69.47. From Equation 69.25, we obtain 


e(t)= ome, (69.60) 
k 


Ee Dee kT on (.), (69.61) 


[Note that, for convenience, we have re-reversed time in comparison with Equation 69.57 and that, from 
Equation 69.25, we have cj = /2kn fo vo(x) sin kx dx—although we will have no need of any explicit 
information about v9.] 

The form of Equation 69.60 is a Dirichlet series; note that this becomes a power series in § = e-™t with 
only the k” powers appearing: e~*** = ext s&h The theory of such series centers on the Miintz-Szasz 
Theorem (extending the Weierstrass Approximation Theorem) which, for our purposes, shows that only 
quite special functions can have L?-convergent expansions (Equation 69.60) when 1/24 < oo. One has 


estimates for Equation 69.60 of the form 


ICkl < Bell Pllz2(0,00) (69.62) 


with the values of B, explicitly computable as an infinite product 


Be = J1+2%] | 


itk 


142d, 
hi Mk 


1+ (69.63) 


(convergent when )7, 1/, is convergent); note that 1/8; is the distance in L?(0, 00) from exp[—,#] to 
span {exp[— jt] : i # k}. L. Schwarz has further shown that for functions given as in Equation 69.60 one 
has 


Illz2(,00) < Mri ¢llzz¢,7)- (69.64) 


Combining these estimates shows that 


Be) 22x? 
I(T, -Yilz2¢0,) $ Cr rilgllae,r) (<i = E eee hs (69.65) 
k 
The sequence Bx increases moderately rapidly as k > oo but the exponentials exp[—k’n”T] decay 
even more rapidly; so the sum giving C7. is always convergent and Equation 69.65 provides a bound 
(|All < P'7Cr < 00) for the observation operator * A: M = Mjo,7] > 1?(Q): ge v(T,-), when T > 0 
is arbitrarily small. 


* We note at this point that an estimate by Borwein and Erdélyi makes it possible to obtain comparable results when U/ 


has the form U = E€ x {0} with € any subset of [0, T] having positive measure; one consequence of this is a proof of the 
bang-bang principle for time-optimal constrained boundary control. 
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A somewhat different way of looking at this is to note that the linear functional: M > R: pr c& must 
be given by a function g, € L7(0, T) such that 


T ae, 
i nide™ dt = buf) wee (69.66) 
; ae 


If we think of g;,(-) as defined on R (0 off [0, T]), we may take the Fourier transform and note that 
Equation 69.66 just asserts the “interpolation conditions” 


&(—jri) = V208;4 G=V—D; (69.67) 


so it is sufficient to construct functions g satisfying Equation 69.67, together with the properties required 
by the Paley-Wiener Theorem to get the inverse Fourier transform in L7(0, T) with ||gx|| = Bx. This 
approach leads to the sharp asymptotic estimate 


In ||A|| = O/T) asT—0, (69.68) 


showing how much more difficult* observability or controllability becomes for small T, even though one 
does have these for every T > 0. 

A variant of this considers an interior point observation ¢(t) := v(t, a). The observability properties 
now depend on number-theoretic properties of 0 < a < 1 — for rational a = m/n, one obtains no infor- 
mation at all about c, when k is a multiple of n, since then sin kaa = 0, and there is difficulty if a is 
approximable by rationals too rapidly. It can be shown that one does have bounded observability (with 
arbitrarily small T > 0) for a in a set of full measure whereas the complementary nullset for which this 
fails is uncountable in each subinterval.’ 

Finally, we note that an essentially identical treatment for all of the material of this subsection would 
work more generally for Equation 69.21 and with other boundary conditions. 


69.5.3 Higher-Dimensional Geometries 


We have already noted that the geometry may be significant here in ways which have no finite-dimensional 
parallel. For higher-dimensional cases, we note first that we can obtain observability for any “cylindrical” 
region Q2 := (0, 1) x QC R4 withu = (0, T) x [0 x Q] by using the method of “separation of variables” 
to reduce this to a sequence of independent one-dimensional problems: noting that we have here 


Ke = ker the ek, 0(x, X) = [V2 sin knx]é(X), 
we obtain 


Avy(-50,-) = ) “LAr gel (xe? Be), 
L 


with @o(t) =e" (ve(t,0,-), 20) 


where A, is the observability operator for Equation 69.59. It is easy to check that this gives ||A|| < ||Ai|| < 
oo and we have nullcontrollability by duality. 

For more general regions, when U/ is all of & := (0, T) x OQ we may shift to the context of nullcontrol- 
lability for Equation 69.55 and rely on a simple geometric observation. Suppose we have 2c Qc R4, 


This may be compared to the corresponding estimate: || A|| = O Ce +1/ 2)) for the finite-dimensional setting, with K, 


the minimal index, giving the rank condition there. 
Since the “bad set” has measure 0, one might guess that observation using local integral averages (as a “generalized 
thermometer”) should always work but, somewhat surprisingly, this turns out to be false. 
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where Q is some conveniently chosen region (e.g., a cylinder, as above) for which we already know 
that we have nullcontrollability for Equation 69.55, that is, with Q, U/ replaced by O = (0, T) x Q and 
U = & = (0,T) x OQ, respectively. Given any initial data up € L?(Q) for Equation 69.55, we extend it as 0 
to all of Q and, as has been assumed possible, let i be a (controlled) solution of Equation 69.55, vanishing 
on all of Q at time T. The operator B acting (at ©) on i will have some value, which we now call “g” and 
using this in Equation 69.55 necessarily (by uniqueness) gives the restriction to Q of u which vanishes 
at T. Thus, this g is a nullcontrol for Equation 69.55. As already noted, once we have a nullcontrol g for 
each upg, it follows, as noted earlier, that the nullcontrol operator C for this setting is well defined and 
continuous; by duality, one also has bounded observability. 

At this point we note an irreversible deep connection [8] between the control theories for the wave and 
heat equations: 


observability for the wave equation wy = Aw for some [82,U] and some T* > 0 implies 
observability for the heat equation u; = Au for arbitrary T > 0 in the same geometric setting 
[Q; U] 


with observability results for the wave equation obtainable [8,10] from known Scattering Theory results. 
From this, one obtains observability/controllability for the heat equation for a variety of geometric settings, 
but there is a price in terms of significant geometric restrictions on [Q, U] related to wave propagation. 
On the other hand, it has now been shown that use of an arbitrary patch U C Q or U C OQ suffices 
for observation or control of Equation 69.1. This uses the technique of Carleman estimates* which is 
quite technical; hence we have given here only a very brief sketch of how it applies to Equation 69.1. 
Given, for example, an interior patch U C Q; so we would be considering the restriction to U as the 
observation operator in Equation 69.57, one begins with a positive function > on Q vanishing on 0Q and 
with |V¢| > 0 outside the patch U and then, with parameters r, s, one sets 
o25bmax — psx) 7s e502) e-te 
~@Mma-ynr °~@na-ynr *~ ye“? ae 


After integration by parts of products coming from relevant derivatives, one obtains an estimate of the 


form 
7 - T 
sf [ [imate iave+ lode + [or] < cy f [ow 
0 Q 0 U 


for large enough s and then r = r(s). From this one can obtain the desired observability estimate ||u7||o < 
C|\u\lzz and so also has nullcontrollability using controls supported in this interior patch. (Further, the 


structure of Equation 69.69 gives the asymptotics Equation 69.68 in this context as well.) 

For boundary control supported in a patch U C 0Q, one can use a trick: extend the region Q to a 
larger Q by a small bulge sitting on U and select a (very small) patch U in the interior of the bulge. 
Applying the patch control result above to the setting (, 0), one can get a nullcontrol supported in U 
for Q; the trace of this solution on the part of U contained in Q (extended as 0 on the rest of AQ) will be 
a nullcontrol, obviously supported on U, for the problem on Q as desired. 

As final comments we note concern for similar problems with nonlinearities in the equation or in 
the boundary conditions. Perhaps still more interesting is concern for systems in which the heat equa- 
tion is coupled with other components. For example, one such (linear) example which has been given 
consideration by several authors is a thermoelastic plate model 


wetAw=add, 0 =Ad—aAw, (69.70) 


coupling an Euler-Bernoulli plate with heat conduction. The interesting problem here is observabil- 
ity/nullcontrollability with observation or control limited not only to a patch U but also to a single 


* The Carleman estimate technique applies not only to the heat equation, but also to similar problems for wave and elliptic 


partial differential equations, and so on; see, for example, [4]. 
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component (temperature J or displacement w or momentum w;). (As of this writing, this is known for 
an interior patch U, but not for a boundary patch since the trick above does not apply when one wishes 
to consider only one component of a system.) 
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70.1 Comparison with the Finite Dimensional Case 


The general question of observability in the finite dimensional setting concerns a system 
x=f(x,...), xER", 


where the ellipsis indicates that the system might involve additional parameters, controls, disturbances, 
etc., and an output (measurement, observation) function 


y=a(x,...), yeR”. 


Observability theory is concerned, first of all, with the question of distinguishability, i.e., whether distinct 
system trajectories x(t), X(t) necessarily give rise to distinct outputs y(t), )(t) over a specified time inter- 
val. The observability question, properly speaking, concerns actual identification of the trajectory x(t), 
equivalently, the initial state x, from the available observations on the system with an ultimate view 
to the possibility of reconstruction of the trajectories, or perhaps the initial or current states, which, in 
application, are generally not directly available from the outputs, typically consisting of a fairly small set 
of recorded instrument readings on the system. In the linear case, observability and distinguishability are 
equivalent, and both questions can be treated in the context of a vector/matrix system 


x= A(t)x, xeER", A(t)eR™”, (70.1) 
together with an output vector y related to x via 
y=C(t)x, yeR™, C(itheR™", (70.2) 


and it is a standard result [1] that observability obtains on an interval [0, T], T > 0, just in case y=0 
implies that x =0 on that interval. This observability property, in turn, is equivalent to the positive 


70-1 
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definiteness of the matrix integral 
T 
Z(T) =[ D(t, 0)*C(t)* C(t) B(t, 0) dt, (70.3) 
0 


where ®(t,s) is the fundamental solution matrix of the system with ®(s,s) =I. The initial state x9 can 
then be reconstructed from the observation y(t) by means of the reconstruction operator (cf. [18]) 


T 
x = 2Z(T)"} i ®(t, 0)*C(t)* y(t) dt. (70.4) 
0 


In the constant coefficient case A(t) = A, observability (on any interval of positive length) is equivalent 
to the algebraic condition that no eigenvector of A should lie in the null space of C; there are many other 
equivalent formulations. 

In the (infinite dimensional) distributed-parameter setting, it is not possible to provide any comparably 
concise description of the general system to be studied or universal definition of the terms involved, but 
we will make an attempt in this direction later in the chapter. To set the stage for that, let us begin with a 
very simple example that illustrates many of the complicating factors involved. 

The term distributed-parameter system indicates a system whose state parameters are distributed over a 
spatial region rather than constituting a discrete set of dependent variables. For our example, we take the 
spatial region to be the interval [0, 1] and represent the state by a function w(x, t), x € [0, 1], t € [0, 00). 
Let us think of w(x, t) as representing some physical property (temperature, concentration of a dissolved 
chemical, etc.) in a fluid moving from left to right through a conduit whose physical extent corresponds to 
0 <x <1 witha uniform unit velocity. If we assume no diffusion process is involved, it is straightforward 
to see that w(x, t) will be, in some sense that we do not elaborate on at the moment, a solution of the 
first-order partial differential equation (PDE) 


Ow Ow _ 


eee 70.5 
ot Bs Ox ue) 

to which we need to adjoin a boundary condition 
w(0, t) = v(t), t > 0, (70.6) 


and an initial condition or initial state given, without loss of generality, at t = 0, 
w(x, 0) = wo(x), x € [0, 1]. (70.7) 


It can be shown (cf. [4], e.g.) that with appropriate regularity assumptions on v(t) and wo(x) there is a 
unique solution w(x, t) of Equations 70.5 through 70.7 for x € [0, 1], t € [0, 00). 

We will consider two different types of measurement. The first is a point measurement at a given 
location xo, 


y(t) = w(x, t), (70.8) 


while the second is a distributed measurement, which we will suppose to have the form 


1 
y= f c(x) w(x, t) dx, (70.9) 
0 


for some piecewise continuous function c(x), defined and not identically equal to zero on [0, 1]. 
Let us suppose that we have an initial state wo(x) defined on [0, 1], while the boundary input (Equa- 
tion 70.6) is identically equal to 0. Let us examine the simplest form of observability, distinguishability, 
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for the resulting system. In this case, the solution takes the form, for t > 0, 


wo(x—t), t<x<l1, 
0.x <t. 


w(x, t)= | 


For a point observation at xo, the output obtained is clearly 


wo(x9—t), O<t< x9, 
y(t) = w(x, t) = 
0,t > xo. 


We see that if xo < 1, the data segment consisting of the values 
wo(x), xo <x<1 


is lost from the data. Consequently, we do not have distinguishability in this case because initial states 
wo(x), Wo(x) differing only on the indicated interval cannot be distinguished on the basis of the observation 
y(t) on any interval [0, T]. On the other hand, for x9 = 1 the initial state is simply rewritten, in reversed 
order, in the values of y(t),0 < ft < 1 and, thus, we have complete knowledge of the initial state wo(x) 
and, hence, of the solution w(x, tf) determined by that initial state, provided the length of the observation 
interval is at least unity. Ifthe length of the interval is less than one, we are again lacking some information 
on the initial state. It should be noted that this is already a departure from the finite dimensional case; 
we have here a time-independent system for which distinguishability is dependent on the length of the 
interval of observation. 

Now let us consider the case of a distributed observation; for definiteness, we consider the case wherein 
c(x) is the characteristic function of the interval [1 — 8, 1] for 0 < 5 < 1. Thus, 


1 min{0,1—t} 
y(t) =) w(x, t) a= [ wo(x) dx 
1 


—$ min{0,1—(t+8)} 


If y(t) = 0 on the interval [0, 1], then by starting with t = 1 and decreasing to t = 0 it is easy to see that 
wo(x) = 0 and thus w(x, t) = 0; thus, we again have the property of distinguishability if the interval of 
observation has length > 1. But now an additional feature comes into play, again not present in the finite 
dimensional case. If we consider trigonometric initial states 


wo(x) =sinwx, w>0 


we easily verify that |y(t)| < 4, t > 0,a bound tending to zero as w — oo. Thus, it is not possible to bound 
the supremum norm of the initial state in terms of the corresponding norm of the observation, whatever 
the length of the observation interval. Thus, even though we have the property of distinguishability, 
we lack observability in a stronger sense to the extent that we cannot reconstruct the initial state from 
the indicated observation in a continuous (i-e., bounded) manner. This is again a departure from the 
finite dimensional case wherein we have noted, for linear systems, that distinguishability/observability is 


equivalent to the existence of a bounded reconstruction operator. 


70.2 General Formulation in the Distributed-Parameter Case 


To make progress toward some rigorous definitions we have to introduce a certain degree of precision 
into the conceptual framework. In doing this we assume that the reader has some background in the basics 
of functional analysis [17]. Accordingly, then, we assume that the process under study has an associated 
state space, W, which we take to be a Banach space with norm ||w||w [in specific instances, this is often 
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strengthened to a Hilbert space with inner product (w, w) w]. The process itself is described by an operator 
differential equation in W, 
w- Aw, (70.10) 


where A is a (typically unbounded, differential) closed linear operator with domain D(A) constituting 
a dense subspace of W satisfying additional conditions (cf. [6], e.g.) so that the semigroup of bounded 
operators e“' is defined for t > 0, strongly continuous in the sense that the state trajectory associated with 
an initial state wo € W, 

w(t) = eo, (70.11) 


is continuous in W for t > 0. Many time-independent PDEs can be represented in this way, with A corre- 
sponding to the “spatial” differential operator appearing in the equation (e.g., in the case of Equation 70.1 
we could take W to be C[0, 1], A to be the operator defined by Aw = —Ow/Ox and D(A) = c [0, 1], the 
subspace of C[0, 1] consisting of continuously differentiable functions on [0, 1] with w(0) = 0). It should 
be noted that the range of et, and hence w(t), is not, in general, in D(A). It can be shown that this is the 
case if wo € D(A). 

Now let Y be a second Banach space, the output, or measurement space and let C: W — Y be the 
observation operator; in general, unbounded but with domain including D(A). Further, given an obser- 
vation interval [0, T], T > 0, let Vr, be the space of observations; e.g., when Y is a Banach space we might 
let Vr = C([0, T]; Y), the space of Y continuous functions on [0, T] with the supremum (Y) norm, or, in 
the case where Y is a Hilbert space, we might wish to take Yr = L?([0, T]; Y), the space of norm square 
integrable functions with range in Y. This space is generally defined so that, for an initial state wo € D(A), 
which results, via Equation 70.11, in a trajectory w(t) € D(A), the observation function is 


y(t) = Cw(t) € Yr. (70.12) 


The observation operator C is said to be admissible if the linear map from D(A) to Yr, wo —> y(-), has a 
continuous extension to a corresponding map from W to yy; we will continue to describe this map via 
Equation 70.12 even though Cw(f) will not, in general, be defined for general wo € W. This whole process 
may seem very complicated but it cannot be avoided in many, indeed, the most important, examples. In 
fact, it becomes necessary in the case of point observations on Equation 70.1 if that system is posed in the 
space W = L?[0, 1] because, although the state w(-, t) is defined as an element of L”[0, 1] for each t, the 
value w(1, t) may not be defined for certain values of t. 
With this framework in place we can introduce some definitions. 


Definition 70.1: 


The linear observed system of Equations 70.10 and 70.12 is distinguishable on an interval [0,T], T > 0, 
if and only if y(-) =0 in Yr implies that wo = 0 in W. 


Definition 70.2: 


Given T > 0 and t € [0, T], the linear observed system of Equations 70.10 and 70.12 is t-observable on 
[0, T] if and only if there exists a positive number y > 0 such that, for every initial state wo € W and 
resulting state (via Equation 70.11) w(t) in W, we have 


IyOlly», = lw@llw, (70.13) 


y(-) being the observation obtained via Equations 70.11 and 70.12. 
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Remark 70.1 


In the finite dimensional context, for t,, t2 € [0, T], t1-observability is equivalent to t2-observability, 
though the (largest) corresponding values of y may be different. We will see, in a heat conduction example 
to be discussed later, that this need no longer be the case for distributed-parameter systems. It does remain 
true for time-reversible distributed-parameter systems of the type discussed here, corresponding, e.g., to 
the wave equation [19], or in the case of the elastic beam equation, which we discuss at some length later. 

Let us note that, just as in the finite dimensional case, the theory of observability for distributed- 
parameter systems forms the basis for observer theory and state estimation theory (cf. [15,18]) in the 
distributed-parameter context. Observability also enters into the question of asymptotic stability for 
certain linear distributed-parameter systems via its connection with the La Salle Invariance Principle 
[12,20]. The question of observability also arises in parameter identification studies [3]. Distributed 
parameter observability plays a dual role to controllability for the dual control system, but the relation 
between the two is not quite as simple as it is in the corresponding finite dimensional context. The reader 
is referred to [5] for details of this relationship. 

In the case of finite dimensional linear systems, a weaker concept than observability, detectability, is 
often introduced. The constant coefficient version of Equations 70.1 and 70.2 is detectable (on the interval 
[0,00)) just in case y(t) =0— x(t) > 0 as t > ov. This concept is not as useful in the distributed- 
parameter context because, unlike the constant coefficient linear case, those components of the solution 
tending to zero do not necessarily tend to zero at a uniform exponential rate (see, e.g., [20]). A more useful 
concept is that of y-detectability for a given y > 0; the system of Equations 70.10 and 70.12 enjoys this 
property on the interval [0, oo) just in case y(t) = O implies that, for some M > 0, ||w(-, t)|lw < Me”, t > 0. 


70.3 Observation of a Heat Conduction Process 


Let us consider an application involving PDEs of parabolic type in several space dimensions. In the 
steel industry, it is important that the temperature distribution in a steel ingot in preparation for rolling 
operations should be as uniform as possible. It is clearly difficult, if not impossible, to determine the 
temperature distribution in the interior of the ingot directly, but the measurement of the surface temper- 
ature is routine. We therefore encounter the problem of the observability of the temperature distribution 
throughout the ingot from the available surface measurements. 

In order to analyze this question mathematically, we first require a model for the process. If we represent 
the spatial region occupied by the ingot as a region Q € R? with smooth boundary I and suppose the 
measurement process takes place over a time interval 0 < t < T, we are led by the standard theory of heat 
conduction to consider the parabolic PDE 


Ow ow Pw ow 
= ; (70.14) 


' ay? * Az? 
where p is the specific heat of the material and k is its thermal conductivity, both assumed constant here. 
The rate of heat loss to the exterior environment is k&, where v denotes the unit normal vector to I, 
external with respect to Q, and this rate is proportional to the difference between the surface temperature 
w at the same point on the surface and the ambient temperature, which we will assume, for simplicity, to 
be zero. The system under study therefore consists of Equation 70.14 together with a Dirichlet-Neumann 
boundary condition 


k— =ow, (70.15) 
where o is a positive constant of proportionality. Using X to stand for the triple x, y, z, the (unknown) 
initial condition is 

w(X, 0) = wo(X). (70.16) 
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The standard theory of parabolic PDEs [22] guarantees the existence of a unique solution w(X,t), X € 
Q,t € [0, T] of Equation 70.14 with the boundary/initial data of Equations 70.15 and 70.16. The available 
measurement data are 


y(X,t)=w(X,t), XeV, te [0,7]. (70.17) 


The question then is whether it is possible to reconstruct the temperature distribution w(X, t), X € Q 
at a particular instant t € [0, T] on the basis of the measurement (Equation 70.18). This question has 
been extensively studied, sometimes indirectly via the dual question of boundary controllability. Brevity 
requirements constrain us to cite only [13], [14] here, but we will indicate below some of the mathematical 
issues involved. 

The Laplacian operator appearing on the right-hand side of Equation 70.14, defined on a dense domain 
in L*(Q) incorporating the boundary condition of Equation 70.15, is known to be a positive self-adjoint 
differential operator with positive eigenvalues \,, k = 1,2, 3, ...and has corresponding normalized eigen- 
functions ;, k = 1, 2, 3,... forming an orthonormal basis for the Hilbert space L?(Q). An initial state wo 
in that space has an expansion, convergent in that space, 


[o,0) 
wo= >> cede. (70.18) 
k=1 
Corresponding to this expansion, the system of Equations 70.14 through 70.16 has the solution 


w(X,t) = D> cee e(X), (70.19) 
k=1 


with the corresponding measurement, or observation 


y(X,t)= D> cee (X), XET, te [0, TI. (70.20) 
k=1 


involving known ix, o%, but unknown cx. 

Let us first consider the question of distinguishability: Can two solutions w(X, t), w(X, t), correspond- 
ing to initial states wo, Wo produce the same observation y(X, f) via Equation 70.17? From the linear homo- 
geneous character of the system it is clear that this is equivalent to asking whether a nonzero initial state 
(Equation 70.18), i.e., such that not all c, = 0, can give rise to an observation (Equation 70.20) that is iden- 
tically zero. This immediately requires us to give attention to the boundary values nx(X, t) = e~**y(X) 
corresponding to wo(X) = ;(X). Clearly, the boundary observation corresponding to a general initial 
state (Equation 70.18) is then 


CO 
y(Xt)= > ocen(Xt), XeT, te [0,7], (70.21) 
k=1 


Can y(X, t), taking this form, be identically zero if the c, are not all zero? The statement that this is 
not possible, hence that the system is distinguishable, is precisely the statement that the n,(X,t),X € 
I’, t € [0, T] are weakly independent in the appropriate boundary space, for simplicity, say L?(T' x [0, T]). 
As a result of a variety of investigations [9], we can assert that this is, indeed, the case for any T > 0. 
In fact, these investigations show that a stronger result, spectral observability, is true. The functions 
nx(X, t) are actually strongly independent in L*(T x [0, T]), by which we mean that there exist biorthogonal 
functions, not necessarily unique, in L?(I x [0, T]) for the n;.(X, t), ie., functions ¢,.(X, t) € L?(V x [0, T]), 
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k =1,2,3,..., such that 
0, k#i, 


70.22 
1, k=j. Oe 


J es. nix) dx dt = 
Px[0,7] 

The existence of these biorthogonal functions implies that any finite number of the coefficients c, can be 
constructed from the observation y(X, t) via 


i i (X, t) y(X, t) dX de. @oa%) 
Tx[0,7] 


Indeed, we can construct a map from the output space, here assumed to be L2(0 x [0, T]), namely, 


K 
SKY =) ec", (X) / t,(X, t) y(X, t) dX dt, (70.24) 
rs Px [0,7] 


carrying the observation Y into the “K-approximation” to the state w(-,t), t € [0, T]. Since the sum 
(Equation 70.18) is convergent in L7(Q), this property of spectral observability is a form of approximate 
observability in the sense that it permits reconstruction of the initial state of Equation 70.18, or a cor- 
responding subsequent state w(-, t), within any desired degree of accuracy. Unfortunately, there is, in 
general, no way to know how large K should be in Equation 70.24 in order to achieve a specified accuracy, 
nor is there any way to obtain a uniform estimate on the effect of errors in measurement of Y. 

The question of t-observability, for t € [0, T], is a more demanding one; it is the question as to 
whether the map S,x defined in Equation 70.24 extends by continuity to a bounded linear map S, 
taking the observation Y into the corresponding state wo. It turns out [21] that this is possible for any 
t > 0, but it is not possible for t = 0; i-e., the initial state can never be continuously reconstructed from 
measurements of this kind. Fortunately, it is ordinarily the terminal state w(-, T) that is the more relevant, 
and reconstruction is possible in this case. The proof of these assertions (cf. [7,21], e.g.) relies on delicate 
estimates of the norms of the biorthogonal functions ¢;.(X, t) in the space L?(T x [0, T]). The boundedness 
property of the operator S, is important not only in regard to convergence of approximate reconstructions 
of the state w(-, t) but also in regard to understanding the effect of an error 8Y. If such an error is present, 
the estimate obtained for w(., t) will clearly be 


W(-, t) = S(Y +8Y) = w(-, 1) + S8Y, (70.25) 


and the norm of the reconstruction error thus does not exceed ||S;||||SY||. A further consequence of 
this clearly is the importance, since reconstruction operators are not, in general, unique, of obtaining a 
reconstruction operator of least possible norm. If we denote the subspace of L7(T x [0, T]) spanned by 
the functions y,(X, f), as described following Equation 70.21, by E(T x [0, T]), it is easy to show that 
the biorthogonal functions ¢; described via Equation 70.22 are unique and have least possible norm if 
we require that they should lie in E(T' x [0, T]). The particular reconstruction operator S,, constructed 
as the limit of operators (Equation 70.24) with the least norm biorthogonal functions ¢,, may then be 
seen to have least possible norm. In applications, reconstruction operators of the type we have described 
here are rarely used; one normally uses a state estimator (cf. [18], e.g.) that provides only an asymptotic 
reconstruction of the system state, but the performance of such a state estimator is still ultimately limited 
by considerations of the same sort as we have discussed here. 
It is possible to provide similar discussions for the “wave” counterpart of Equation 70.14, ie., 


- ew Pw & 
| We we, z). (70.26) 


ax? * dy? * Az? 


with a variety of boundary conditions, including Equation 70.15. A very large number of such studies 
have been made, but they have normally been carried out in terms of the dual control system [19] rather 
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than in terms of the linear observed system. In some cases, methods of harmonic analysis similar to those 
just described for the heat equation have been used [8], but the most definitive results have been obtained 
using methods derived from the scattering theory of the wave equation and from related methods such 
as geometrical optics [2] or multiplier methods [11]. These studies include treatment of cases wherein the 
observation/(dual) control process is restricted to a subset ) C I having certain geometrical properties. 
Other contributions [21] have shown the study of the wave equation to be pivotal in the sense that results 
for related heat and elastic processes can be inferred, via harmonic analysis, once the wave equation results 
are in place. 


70.4 Observability Theory for Elastic Beams 


There are several different models for elastic beams, even when we restrict attention to small deformation 
linear models. These include the Euler-Bernoulli, Rayleigh and Timoshenko models. The oldest and most 
familiar of these is the Euler-Bernoulli model, consisting of the PDE 


Ow _ oO 


( =r) 
(x) —~ | El(x)—~— ], xe€[0,L], t €[0,0), (70.27) 
Ox? 


at? Ax? 

wherein p(x) denotes the mass per unit length and EI(x) is the so-called bending modulus. We are 
concerned with solutions in a certain weak sense, which we will not elaborate upon here, corresponding 
to a given initial state 


w(-,0) = wo € H7[0, L], aes 0) = v9 € L’[0, L], (70.28) 


where H”[0, L] is the standard Sobolev space of functions with square integrable second derivatives on 
the indicated interval. Additionally, one needs to give boundary conditions at x = 0 and at x = L; these 
vary with the physical circumstances. For the sake of brevity, we will confine our discussion here to the 
cantilever case, where the left-hand end is assumed “clamped” while the right-hand end is “free”; the 
appropriate boundary conditions are then 


0 
HOF) =0, —OD=0, (70.29) 
Ox 
Ow 2 
Ox? 


0 O-w 
(L, t) =0, 5 (100) (L, t) =0. (70.30) 


In many applications a mechanical structure, such as a manipulator arm, is clamped to a rotating base that 
points the arm/beam in various directions in order to carry out particular tasks. Each “slewing” motion 
results in a degree of vibration of the structure, which, for most practical purposes, can be thought of as 
taking place within the context of the model of Equations 70.27 through 70.30. In order to attenuate the 
undesired vibration, it is first of all necessary to carry out an observation procedure in order to determine 
the oscillatory state of the system preparatory to, or in conjunction with, control operations. A number of 
different measurement options exist whose feasibility depends on the operational situation in hand. We 
will cite three of these. In the first instance, one might attach a strain gauge to the beam near the clamped 
end. The extension or compression of such a (normally piezoelectric) device provides a scaled physical 
realization of the mathematical measurement 


oe? 
y(t) = 70, t). (70.31) 


Alternatively, one can place an accelerometer near the free end of the beam to provide a measurement 


equivalent to 
2 


) 
y(t) = il t), (70.32) 
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or one can use a laser device, relying on the Doppler effect, to measure 
Ow 
t) = —(L,t). 70.33 
MO= 3b) (70.33) 


Each of these measurement modes provides a scalar valued function y(t) carrying a certain amount of 
information on the system state w(x, t); the problem, as before, is to reconstruct the initial state, or the 
current state at time T, from the record y(t), 0 < t < T. The mathematical theory of this reconstruction 
is in many ways similar to the one we have just described for the heat equation, but with some significant 
differences. The most notable of these is immediately apparent from the equation itself; it is invariant 
under reversal of the time direction. This means that there is no inherent difference between initial and 
terminal states or, indeed, any intermediate state. We should expect this to show up in the mathematics, 
and it does. 
The differential operator defined by 


Os aw 
Aw) = —-—,7 | El(x)—— }, 70.34 
(Aw) =o ( wo) (70.34) 
on the subspace of H*[0,L] resulting from imposition of the cantilever boundary conditions, is an 
unbounded positive self-adjoint operator with positive eigenvalues, listed in increasing order, \;,k = 
1,2,3,.... The corresponding normalized eigenfunctions ;(x) form an orthonormal basis for L7[0, L]. 
Defining w, = 4; the solution of Equations 70.27 through 70.30 takes the form 


[oe 


w(x, t) = So (cpel*! + dye !*) d(x), (70.35) 
k=1 


where, with wo and vo as in Equation 70.28 


CO (oe) 
wo(x) =) wokde(x), volx) = D> vondx(x), (70.36) 
k=1 k=1 
with 
Wok =Ckt+dks Vo,k = iw, ch — dy). (70.37) 


The norm of the state wo, vo in the state space H 210, L] ® L7[0, L] is equivalent to the norm of the double 
sequence {wo,k; Vo,k} in the Hilbert space, which we call ee That norm is 


1 
ee) 2 
IlWo,4> Yoel = bs (@KWoy + 7) : 


k=1 


Any one of the measurement modes discussed earlier now takes the form 


a) 
y(t) = y (vecne’* + S¢dxe Pk") Z (70.38) 
k=1 


with y; and 8; depending on the particular measurement mode being employed. Thus, in the cases of 
Equations 70.31, 70.32, and 70.33, respectively, we have 


ao 
Ve = Oe = “2 (70.39) 
Vk = 8k = de oe(L), (70.40) 


Vk =ioxox(L), 8 = —imgeg(L). (70.41) 
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Just as in the earlier example of the heat equation, but now we are concerned only with the scalar 
exponential functions 


elke ptt k = 1,2,3,..., (70.42) 


everything depends on the independence properties of these functions in L?[0, T], where T is the length 
of the observation interval, in relation to the asymptotic growth of the w;, k — oo, the basic character of 
which is that the w, are distinct, increasing with k, if ordered in the natural way, and 


OK = O(k*), k > 00. (70.43) 


The relationship between properties of the functions of Equation 70.42 and the asymptotic and/or 
separation properties of the exponents w,; is one of the questions considered in the general topic of 
nonharmonic Fourier series, whose systematic study began in the 1930s with the work of Paley and 
Wiener [16]. The specific result that we make use of is due to A. E. Ingham [10]. Combined with other, 
more or less elementary, considerations, it implies that ifthe , are real and satisfy a separation condition, 
for some positive integer K and positive number I’, 


Okt] [OR+T, k= K, (70.44) 


then the functions of Equation 70.42 are uniformly independent and uniformly convergent in L7[0, T], 


provided that T > 2%, which means there are numbers b, B, 0 < b < B, such that, for any square summable 
sequences of coefficients cy, dy, k = 1,2, 3,..., we have 
CO [o,@) [o,@) 
b>” (lel? del?) < SS (cxel Hedge) Moray SBD, (eel? +14el’) (70.45) 
k=1 k=1 k=1 


Since the asymptotic property of Equation 70.43 clearly implies, for any P > 0, that Equation 70.44 is 
satisfied if K = K(I) is sufficiently large, inequalities of Equation 70.45, with b = b(T), B = B(T), must 
hold for any T > 0. It follows that the linear map, or operator, 


CO 
Cr > (cpel* + dye FH) — {cy, dylk = 1,2,3,...) 2, (70.46) 
k=1 


defined on the (necessarily closed, in view of Equation 70.45) subspace of L?[0, T] spanned by the expo- 
nential functions in question, is both bounded and boundedly invertible. Applied to the observation 
y(t), t € [0, T], corresponding to an initial state of Equation 70.36, the boundedness of C, together with 
the relationship of Equation 70.37 between the wo, Vo,, and the c;,, dy, and the form of the e norm, it is 
not hard to see that the boundedness of the linear operator C implies the existence of a bounded recon- 
struction operator from the observation (Equation 70.38) to the initial state (Equation 70.36) provided 
the coefficients 8, yz in Equation 70.38 satisfy an inequality of the form 


Ick] = Cox, |dk| = Dog, (70.47) 


for some positive constants C and D. Using the trigonometric/exponential form of the eigenfunctions 
x(x), one easily verifies this to be the case for each of the observation modes of Equations 70.39 through 
70.41; indeed, the condition is overfulfilled in the case of the accelerometer measurement (Equation 70.40). 
Thus, bounded observability of elastic beam states via scalar measurements can be considered to be typical. 
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At any later time t = 1, the system state resulting from the initial state of Equation 70.36 has the form 


CO 
1.., 
wr(x) = Y > (cos wtwo,k + oi sin xtVo,k) x(x), 
k=1 ‘ 


Me 


V_(x) = (—a, sin wptwo,~ + Cos MEV k) DE (x). (70.48) 


ke] 


Using Equation 70.48 with the form of the norm in @2,, one can see that the map from the initial state 
wo, Vo € H2[0, L] ® L2[0, L] to wz, vz in the same space is bounded and boundedly invertible; this is 
another way of expressing the time reversibility of the system. It follows that the state w,,v, can be 
continuously reconstructed from the observation y(t), t € [0, T] in precisely the same circumstances as 
we can reconstruct the state wo, vo. 
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Aerodynamic fluid flows, flexible structures, plasmas in electromagnetic fields, acoustic and optical waves, 
thermal dynamics, and chemical processes—distributed parameter systems—are so ubiquitous in control 
applications that they are the norm, rather than an exception. Although finite-dimensional techniques 
often suffice in solving such control problems, there are significant advantages in considering such systems 
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in their full partial differential equation (PDE) format, which has driven the development of the field of 
control of distributed parameter systems since the 1960s. 

Numerous papers, surveys, and even books exist on control of distributed parameter systems [2,7,8, 
17,18], dedicated to solving problems of controllability and optimal control. The latter are particularly 
of interest as they produce feedback laws, which afford a degree of robustness to various uncertainties. 
Optimal control laws for PDEs require solutions to operator Riccati equations—infinite dimensional 
nonlinear equations which are in general very difficult to solve. In this chapter we present the recently 
emerged approach, inspired by the finite-dimensional “backstepping” design for nonlinear systems [12], 
which avoids the requirement to solve Riccati equations. Instead of Riccati equations, our design problem 
consists in solving only a linear hyperbolic PDE in the spatial variables, which is performed once, offline. 
This simplification of the design problem is achieved by foregoing optimality. The approach consists 
in constructing a “coordinate transformation” and a control law, to cast the closed-loop system in a 
desirable form, which we refer to as the “target system.” This is similar to the approach employed in 
feedback linearization and backstepping. The choice of a target system is different for each of the different 
classes of PDEs. This factor requires a deeper level of involvement on the part of the control designer, 
who needs to have a certain level of insight into stability properties of different classes of PDEs. However, 
for this additional analytical effort, the designer is rewarded with a control law which is much simpler 
than optimal controllers employing Riccati equations, and with a dynamic behavior which is physically 
intuitive because it follows the behavior of well-understood classes of physical PDEs. 

This line of research at present deals only with the problems of boundary control. Many solutions 
exist, at least in principle, for control of PDEs both with interior or distributed controls, as well as 
with boundary controls. It is, however, generally recognized that boundary control problems, where the 
actuation is applied only through the boundary conditions of a PDE, are not only the most realistic in 
many applications (e.g., this is the only way to control fluid flows using micro-electro-mechanical systems 
[MEMS] actuators), but are analytically the most challenging because the input and output operators are 
unbounded. On the intuitive level, the boundary control problems are always characterized by the state 
space having a spatial dimension at least one higher than that of the actuator(s). For instance, the system 
evolving over a rectangle and controlled only through the sides of the rectangle. In boundary control 
problems one necessarily has an order of magnitude fewer control inputs than the states. Despite this 
“extreme” character of the boundary control problem, the backstepping approach happens to be perfectly 
suited for it as it is based on propagating a control action through a sequence of ordered subsystems [12]. 
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To introduce the basic idea of the backstepping control design, let us start with the simplest unstable PDE, 
reaction-diffusion equation 


uz(x, t) = Ux (x, t) + u(x,t), x € (0,1) (71.1) 
u(0, t) = 0, (71.2) 
u(1,t) = U(t), (71.3) 


where } is an arbitrary constant and U(t) is the input. The open-loop system (Equations 71.1 and 71.2) 
(with U(t) = 0) is unstable with arbitrarily many unstable eigenvalues for sufficiently large d. 
The main idea of the backstepping method is to use the coordinate transformation 


w(x, t) = u(x, t) — b k(x, y)u(y, t) dy (71.4) 
0 


and the control law 


1 
U(t) = [ k(1,y)uly,t) dy (71.5) 


Boundary Control of PDEs 71-3 


to transform the systems (Equations 71.1 through 71.3) into the heat equation 


wz(x, t) = Wyx(x, t) (71.6) 
w(0, t) = 0, (71.7) 
w(1,t) = 0, (71.8) 


which is exponentially stable. 
To derive the equation for k(x, y) we first substitute Equation 71.4 into Equation 71.6: 


w(x, t) = u(x, t) — [ eee y)(Uyy(y, t) + uly, £)) dy 
= Use (x, t) + du(x, t) — k(x, x)ux(x, t) + k(x, O)ux(0, t) + ky (x, x)u(x, £) 
= [ se: y) + k(x, y))u(y, t) dy, (71.9) 
Wyoc(X, t) = Use (2x, t) — k(x, x) uc (x, #) — ker xe, x) (ee, £) 


_ ae t)— is Kxx(x, y)u(y, t) dy. (71.10) 
dx 0 


Combining Equations 71.9 and 71.10 and using Equation 71.1 gives 
0 = wi(x, t) — Wax(x, t) + cw(x, t) 
= i {Kee 9) — yy x,y) — DACs, y)} 09,8) dy 
0 


+ (2k s+ ) u(x,t) + k(x, 0)tt<(0, 1). (71.11) 


We can see now that the kernel k(x, y) must satisfy the following hyperbolic PDE: 


Kyx(x,¥) — kyy(x, y) =hk(x,y), (ay) ET, (71.12) 
k(x, 0) = 0, (71.13) 
Ke,x) =~. (71.14) 


where J ={x,y : 0<y<x<]1}. The solution to this PDE is [21] 


q (Vx@?=)) 


k(x, y) = (71.15) 
y y NG) 
where I is a first-order modified Bessel function. The controller (Equation 71.5) is therefore, 
pfs ys (71.16) 
U(t) = if iN u(y, t) dy. 71.16 
P y 1d) yt) ay 
One can show that the transformation (Equation 71.4) is invertible and its inverse is 
x fi (v d(x? =) 
u(x, t) = w(x, t) / hy w(y,t) dy. (71.17) 
0 d(x? — y?) 


where J; is the usual (nonmodified) Bessel function of the first order. The smoothness of the kernels of 
the direct and inverse transformations in x and y establishes the equivalence of norms of u and w in both 
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L? and H!. Therefore, from the properties of the heat equations 71.6 through 71.8 we conlude that the 
closed-loop system is exponentially stable in L? and H?. 

Note that the systems (Equations 71.1 through 71.3 and 71.16) are not only well posed but their 
solutions are explicitly available. To find this solution, we first solve the heat equation (Equations 71.6 
through 71.8): 


o° 1 
w(x, t)=2 > emt sin(1tx) i} wo(&) sin(mné) dé. (71.18) 
n=1 0 


The initial condition wo can be calculated explicitly from up via Equations 71.4 and 71.15. Substituting 
the result into Equation 71.17, changing order of integration, and calculating one of the integrals we 
obtain the explicit solution to closed-loop systems (Equations 71.1 through 71.3 and 71.16): 


(oe) 


23,2 : 
ws) = Doe" eons) [nC G)uol®) (71.19) 
0 


n=1 


where 
) Y 1 - q, (v (EZ _ *)) ; 
Wn x) = sin( tnx +f * Toe) j sin(1n§) dé, 


2mn 
———— sin( Vr_+ 1 n?x) ; (71.20) 
Vi+ 12 n2 ( 


The above control design is easily extended for the Neumann type of actuation, one only needs to 
replace the boundary condition (Equation 71.8) with the Neumann one. The control law is 


n(x) = 


1 
ux(1, t) = ve) =k Duct) + f k<(1,y)u(y, t) dy, (71.21) 
0 


where k(x, y) is given by Equation 71.15. 


For the rest of the chapter we will not repeat the procedure of derivation of a PDE for the kernel and 
the arguments of the previous paragraph proving stability and well posedness. 


71.3 Observers 


The stabilizing controller developed in the previous section requires complete measurements from the 
interior of the domain which are usually unavailable. So we look for the observers that estimate u(x, t) 
inside the domain. Consider the plant 


Ut(x, f) = U(x, t) + Ku(x, f), (71.22) 
ux(0, t) = 0, (71.23) 
u(1, t) = U(t). (71.24) 


Suppose the only available measurement of our system is at x = 0, the opposite end to actuation. To 
estimate the state inside the domain, we introduce the following observer: 


tt (x, t) = ty x(x, t) + XU(x, t) + pi(x)[u(O, t) — (0, t)], (71.25) 
a(1,t) = U(t). (71.27) 


Here p; (x) and pio are output injection functions (pj9 is a constant) to be designed. Note that we introduce 
output injection not only in Equation 71.25 but also at the boundary where the measurement is available. 
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The observer (Equations 71.25 through 71.27) is in the standard form of “copy of the system plus 
injection of the output estimation error.” This standard form allows us to pursue duality between the 
observer and the controller design, similar to the way duality is used to find the gains of a Luenberger 
observer based on the pole-placement control algorithm, or to the way duality is used to construct a 
Kalman filter based on the LQR design. 

The observer error i! = u — it satisfies the PDE 


itt (x, t) = ityx (x, t) + ia(x, t) — pi(x)i(0, t), (71.28) 
u,x(0, t) = —piqu(0, t), (71.29) 
iu(1, t) = 0. (71.30) 


Observer gains pi (x) and pio should be now chosen to stabilize the error system. We look for a coordinate 
transformation 


u(x, t) = w(x, t) =f p(x y)wly, t) dy (71.31) 
0 
that transforms the system (Equations 71.28 through 71.30) into the exponentially stable system 
Wz(x, t) = Wex (x, t), (71.32) 
w,(0, t) = 0, (71.33) 
w(1, t) =0. (71.34) 


By substituting Equation 71.31 into Equations 71.28 through 71.30 we obtain a set of conditions on the 
kernel p(x, y) in the form of the hyperbolic PDE 


Pyy(% y) — Pxx(x, y) = p(x, y) (71.35) 
for0<y <x < 1, with the boundary conditions 
d x 
Bike givens 71.36 
qb =>5 (71.36) 
p(l,y) =0. (71.37) 
and observer gains given by 
pilx) = ep,(x,0), pio = p(0, 0). (71.38) 
Let us make a change of variables 
x=l-y, y=l—x, p(x y)=pl(xy). (71.39) 


It can be verified that in these new variables the problems (Equations 71.35 through 71.37) become exactly 
the same as Equations 71.12 through 71.14 for k(x, y), and therefore, 


h (Vx@ =F) 


(x,y) = (71.40) 
POY y i) 
Using Equation 71.38 we obtain the observer gains 
(1 — x) nN 
— — J. nx(2 — F =--_-, 71.41 
pi)= 5h (Vie@—9), pio =—3 (71.41) 


The design procedure presented above is easily extended to other input/output setups, such as 
Neumann sensing (u,(0, f) is measured) and sensor/actuator located at the same boundary (e.g., u(1, t) is 
measured, u,(1, t) is controlled); see [22] for details. 

Since the observer design is dual to control design, only control designs will be presented in the rest of 
the chapter. 
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71.4 Other Parabolic Plants 


71.4.1 Reaction—Advection—Diffusion Systems 


Consider the reaction—advection—diffusion PDE 


uz(x, t) = €(x) Ux, (x, t) + B(x) ux (x, t) + A(x) u(x, t), (71.42) 
ux(0, £) = —qu(0, t), (71.43) 
u(1, t) = U(t). (71.44) 


This equation describes a variety of systems with thermal, fluid, and chemically reacting dynamics. The 
spatially varying coefficients come from applications with nonhomogenous materials, unusually shaped 
domains, and can also arise from the linearization. 

Using the so-called gauge transformation, it is possible to convert this system into the one with constant 
diffusion and zero advection terms. Consider a coordinate change 


x 


HS) ds 
v(z,t) =e /4(x)u(x, Heo? , (71.45) 
where ‘ 
* ds lds \~ 
v6 Fay @=(f Faq) - as 
Then the new state v satisfies the following PDE: 
vz(z, t) = EqVzz(Z, t) + X0(z)v(z, t), (71.47) 
vz(0, t) = —qov(0, t), (71.48) 
; Hs) gs 
vayt)= Ue (yet = (71.49) 
where 
e"(x) B(x) 3 (e'(x)) 1 D(x)e’(x) 1:7 (x) 
MS Mes 4 2 16 &(x) a) (x) 4 s(x)’ ve) 
e(0) b(0) e’(0) (71.51) 


Os; = Dy epe) Alege) 


We use the transformation (Equation 71.4) (with x replaced by z) to map the modified plant into the 
target system 


wz(Z, t) = €9Wzz(z, t) — cw(z, t), (71.52) 
w-(0, t) = w(1, t) =0. (71.53) 


Here c is a design parameter that determines the decay rate of the closed-loop system. The transformation 
kernel is found as a solution of the following PDE: 


(z,y), (71.54) 


ho(y) +c 
kexl2,y) — kyy(Z,y) = se 


ky(z, 0) = —qok(z, 0), (71.55) 


1 Zz 
Kez) = —qo- 5— / (nol y) + 0) dy. (71.56) 
0 
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This PDE is well posed but cannot be solved in closed form. However, it is easy to solve it symbolically (by 
successive approximations series) or numerically, substantially easier than solving a Ricatti equation [21]. 
Note that for constant ¢, b, X, and q, the kernel PDE can be solved explicitly. The controller is 


: Y Seq ds — pt bo 
= 1/4 NSO —3/4 i 2e(s ds 
U(t)=e over f k (./ a ) € (ye O”™ u(y, t) dy. (71.57) 
71.4.2 Other Spatially Causal Plants 


Consider the spatially causal system 


ur(x, t) = Uxx(x, t) + g(x) u(0, t) +| f(x, y)uly, t)dy, (71.58) 
0 


where the boundary conditions are u,(0,t) = 0 and u(1, t) = U(t). This equation is partly motivated by 
the model of unstable burning in solid propellant rockets [6]. Another reason to consider this plant is that 
it often appears as a part of the control design for more complicated systems (see Section 71.8). Without 
the derivation we present the PDE for the kernel of the transformation (Equation 71.4) that maps it into 
the heat equations 71.6 through 71.8 [21]: 


Keyxe(x, Y) — kyy (x,y) = -fosy)+ | k(x, )f (8 yd, (71.59) 
y 
ky(x, 0) = g(x) — i k(x, y)g(y)dy, (71.60) 
0 
k(x, x) =0. (71.61) 


This PDE can be solved using the method of successive approximations. The controller is given by 
Equation 71.5. 


71.5 Complex-Valued and Coupled PDEs 


The design scheme presented in previous sections extends to complex-valued PDEs in a straightforward 
way. The only difference is that the gain of the transformation (Equation 71.4) becomes complex-valued. 
As an example, consider the linear Schrédinger equation 


Uz(x, t) = —juxx(x, f), (71.62) 
ux(0, t) = 0, (71.63) 
u(1, t) = U(t). (71.64) 


Note that u(x,t) is a complex-valued function so that Equations 71.62 through 71.64 can be viewed 
as two coupled real-valued PDEs. Without control, this system displays oscillatory behavior and is not 
asymptotically stable. Let us think of —j as the diffusion coefficient in the heat equation and apply the 
transformation (Equation 71.4) to map the plant into the target system 


w(x, t) = —jwxx(x, t) — cw(x, t), (71.65) 
wx(0, t) = w(1, t) = 0. (71.66) 
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It is easy to check that this system is well damped for c > 0. The gain kernel for the controller (Equa- 
tion 71.5) is found as 


6 (Vac =7)) 
Vc — y*) 


“laa [c+ spe ( e(1= 98) ) +G— ben ( <(1-»))]. (71.67) 


where ber (-) and beij(-) are Kelvin functions. The controller is given by (Equation 71.5). 

A more general version of the Schrédinger equation is a Ginzburg-Landau equation, which is a 
popular simplified model for studying vortex shedding in the flows past submerged obstacles. In [1], 
a flow around a two-dimensional circular cylinder was considered and the backstepping method was 
successfully applied to design the output-feedback boundary controllers with both sensing and actuation 
on the cylinder surface. 

Another application which involves dealing with coupled equations is a thermal convection loop [5], 
which describes the dynamics of a heated fluid between two concentric cylinders in a gravity field. In [27], 
the backstepping design was combined with the singular perturbation approach to solve this problem. 


Ky) =-c 


71.6 First-Order Hyperbolic Systems 


We now turn our attention to hyperbolic PDEs. Before considering oscillatory systems such as strings and 
beams we deal with first-order hyperbolic equations. These equations describe quite different physical 
problems such as traffic flows, chemical reactors, and heat exchangers. They can also serve as a model for 
delays. 

The general first-order hyperbolic equation that can be handled by the backstepping method is 


uy(x, t) = ux(x, t) + Au(x, t) + g(x)u(0, t) + [ fosnuv. t) dy, (71.68) 
0 


u(1, t) = U(t). (71.69) 


Note that unlike in second-order (in space) PDEs, here we specify only one boundary condition. For g or 
f positive and large, the uncontrolled plant is unstable. 

Following our general procedure, we use the transformation (Equation 71.4) to convert the plant 
(Equation 71.68) into the target system 


wz(x, t) = wx(x, £); (71.70) 
w(1, t) =0. (71.71) 
This system has the solution 
_ Jwolt+x) O<t<1, 
w(x,t) = {; ij (71.72) 


where wo(x) is the initial condition. We see that this system acts as a pure delay and converges to zero in 
finite time. 
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One can derive the following well-posed kernel PDE from Equations 71.68 through 71.71: 


ky (x,y) + ky(x, y) =i k(x, &)f (5, y)dé — f(x,y), (71.73) 
yy 

k(x, 0) = / k(x, y)g(y)dy — g(x). (71.74) 
0 


This PDE has a closed-form solution for constant g and f but needs to be solved numerically in general. 
The controller is given by Equation 71.5. 


71.7 Wave Equation 


71.7.1 Undamped Wave Equation 


Consider the PDE that describes a vibrating string on a finite interval 


Upp (x, t) = Uxx(x, t), (71.75) 
ux(0, t) = 0, (71.76) 
u(1,t) = 0. (71.77) 


The boundary conditions correspond to the situation where the end of the string at x = 1 is “pinned” and 
the other end is “free.” 

One classical method of stabilizing this system is to add a damping term to the boundary. Specifically, 
the boundary condition (Equation 71.76) becomes 


ux(0, tf) = cou; (0, t). (71.78) 


The boundary control provided by Equation 71.78 has the capacity to add considerable damping to the 
system, however it requires actuation on the free end x = 0, which is not always feasible. 
Let us now consider the wave equation with damping boundary control at the constrained end 


Un(x, t) = Uxx(X, t) (71.79) 
ux(0, t) = 0 (71.80) 
ux(1, t) = U(t). (71.81) 
The attempt to use the controller U(t) = —c,u;(1, t), c) > 0, fails because in that case the system has an 


arbitrary constant as an equilibrium profile. To deal with this multitude of equilibria a more sophisticated 
controller at x = 1 (e.g., backstepping) is needed if the boundary condition at x = 0 is to remain free. 
The backstepping transformation 


w(x, t) = u(x, t) +c i u(y, t) dy (71.82) 
0 


maps Equations 71.79 through 71.81 the plant into the target system 


Wit (X, tf) = Wax (x, f), (71.83) 
w,(0, t) = cow(0, t), (71.84) 
wx(1,t) = —cyw;(1, £). (71.85) 


The idea is to use a large cg in the boundary condition at x = 0 to make w,(0, t) = cow(0, t) behave like 
w(0,t) = 0. 
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To establish stability of the target system, consider the Lyapunov function 


1 1 
VS (Uo? + lvl? + cw?) +8 J (1+ x) sled) ae (71.86) 
0 
One can show that for sufficiently small 8, there exist 7, mz > 0 such that 
mU<V<mU, U=|lwell? + |lwell? +70), (71.87) 
which shows that V is positive definite. Straightforward calculation gives 
; iy 8 
V = —(c1 —8(1 +7) we) — ; [sl]? + lwell?] — 5 (w2(0) + c3w?(0)) , (71.88) 


which is negative definite for sufficiently small 8 < ee: . From Equation 71.87 it follows that 
if 


U(t) < Me~*/MU(0) 


for some (possibly large) M, and therefore the target system is exponentially stable. 
The resulting Neumann backstepping controller is given by 


1 
U(t) = —cy uz (1, t) — cou(1, t) — ae | ur(y, t)dy, (71.89) 
0 


where co is large and c; is around 1. Some insight into the properties of the above controller may 
be obtained by observing the individual terms in Equation 71.89. The term —c,u;(1,t) provides basic 
damping and the action of —c,u;(1, ft) — cou(1, ft) is similar to PD control. The last term is a spatially 
averaged velocity and is the backstepping term that allows actuation at the constrained end. 


71.7.2 Wave Equation with Kelvin— Voigt Damping 


A totally different backstepping design is possible when the wave equation has a small amount of “Kelvin- 
Voigt” damping (internal material damping, present in all realistic materials): 


Ur (x, t) = el + dO; )Uxx (x; t), (71.90) 
ux(0, t) = 0, (71.91) 
u(1, t) = U(E), (71.92) 


where 0; is the time derivative and d is a small positive constant. Using the usual backstepping transfor- 
mation (Equation 71.4) with the kernel 


ff ( c(x2 =~) 
k(x, y) = —cx (71.93) 


and the controller 


1 (vea=y) 
U(t) = / Cc u(y, t) dy, (71.94) 
0 


Jc(1 — y?) 


we map the plant into the target system 
Wet (x, t) = (1 + dds) (Wax (x, t) — cw(x, t)), (71.95) 


w,(0, t) = 0, (71.96) 
w(1,t) =0. (71.97) 
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The nth pair of eigenvalues o,, of this system satisfies the quadratic equation 


2 2 
p+ dler (+20) |ont [e+ (5 t nn) |=0 (71.98) 
where n = 0, 1, 2,.... There are two sets of eigenvalues: for lower n’s the eigenvalues reside on the circle 
1\? es! 
Re(on) + 5 } + (Im(on))" = a, (71.99) 


and for higher n’s the eigenvalues are real, with one branch accumulating toward —1/d as n > oo and the 
other branch converging to —oo on real axis. Increasing c moves the eigenvalues along the circle in the 
negative real direction and decreases the number of them on the circle (ultimately they become real). This 
increases the damping ratio of the nth conjugate complex eigenvalue pair of the closed-loop system by a 
factor of a 1+ S EETy? c and moves it leftward in the complex plane by a factor of (1 + FEET c) . With 
avery high value of c all of the eigenvalues can be made real. While possibly, this would not necessarily be a 
good idea, neither for servo response, nor for disturbance attenuation, and certainly not from the point of 
view of control effort. Thus, the flexibility to improve the damping using the backstepping transformation 
and controller should be used judiciously, with lower values of c if d is already relatively high. 


71.8 Beams 
The most general 1D beam model—the Timoshenko model—can be represented by the two coupled wave 
equations 
EUpt = Uxx — Ox, (71.100) 
WE = EOxx + (ux — Q), (71.101) 


where u(x, tf) denotes the displacement, a(x, t) denotes the angle of rotation, and the positive constants 
€,,d, and a represent physical parameters of the beam. We consider a beam which is free at the end 
x = 0, that is, 


ux(0, t) = 0(0, t), (71.102) 
a,(0,t) = 0 (71.103) 

and which is controlled at the end x = 1 through the boundary conditions u,(1, t) and a(1, f). 
For the case of small 1, we obtain the so-called “slender beam.” When \ is set to zero, the Timoshenko 


equations 71.100 and 71.101 reduce to the “shear beam” model which we will consider in this section: 


EU = Uxy — Ax, (71.104) 
0=d¢, —V’a+ buy, (71.105) 


where b = ,/a/e. The solution to the a ODE (Equation 71.105) is 
a(x, t) = cosh(bx)a(0, t) + b sinh(bx)u(0, t) — b? i cosh(b(x — y))u(y, t) dy. (71.106) 
0 


The boundary value «(0, f) can be expressed in terms of a(1,t) by setting x = 1 in Equation 71.106. In 
order to eliminate the nonstrict feedback integral term (and thus set «(0, tf) to zero), let us choose our first 
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control to be 


1 
a(1,t) = U,(t) = bsinh(b)u(0, t) — Pf cosh(b(1 — y))u(y, t) dy. (71.107) 
0 


Substituting Equation 71.106 with «(0, t) = 0 into Equation 71.104 gives 
EU (X, t) = Uxy(x, t) + b?u(x, t) — b? cosh(bx)u(0, t) + b° i sinh(b(x — y))u(y, t) dy, (71.108) 
0 


where this PDE is now of the form similar to Equation 71.58. Now consider the transformation (Equa- 
tion 71.4) to the exponentially stable target system 


Ew (X, t) = Wax(x, f), (71.109) 
w,(0, t) = cow(0, t), (71.110) 
wx(1, f) = —c,w;(1, t), (71.111) 


where cg and c; are design parameters. The transformation kernel can be shown to satisfy the following 
PDE 


Kee (xs ¥) = kyy(x, y) + 0? k(x, y) — b? sinh(b(x — y)) + B° ij : k(x, £) sinh(b(é — y))dé, (71.112) 
y 


ky(x, 0) = 7 i k(x, y) cosh(by)dy — b? cosh(bx), (71.113) 
b2 
K(x, x) = — = x— €9. (71.114) 


The second boundary controller (the first one is given by Equation 71.107) is obtained by differentiating 
the transformation (Equation 71.4) with respect to x and setting x = 1: 


b2 1 
ux(1, t) = Up(t) = (5 «) uae) kx(1,y)u(y, t) dy 
0 


1 
-au(it+e f K(1, y)ur(y, t) dy. (71.115) 
0 


71.9 Adaptive Control of Parabolic PDEs 


In systems with thermal, fluid, or chemically reacting dynamics, which are usually modeled by parabolic 
PDEs, physical parameters are often unknown. Thus a need exists for adaptive controllers that are 
able to stabilize a potentially unstable, parametrically uncertain plant. While adaptive control of finite- 
dimensional systems is a mature area that has produced adaptive control methods for most LTI systems 
of interest [10], adaptive control techniques have been developed for only a few of the classes of PDEs 
for which nonadaptive controllers exist. The prebackstepping results [3,4,9] focus on model reference 
adaptive control (MRAC) type schemes and the control action distributed in the PDE domain; see [11] 
for a more detailed literature review. The backstepping controllers presented in previous sections are 
explicitly parametrized in physical parameters of the corresponding plants, making them an ideal choice 
for nominal controllers in adaptive schemes. In the rest of the chapter we overview three different 
design methods based on those explicit controllers—the Lyapunov method, and certainty equivalence 
approaches with passive and swapping identifiers. For tutorial reasons, the presentation proceeds through 
a series of one-unknown-parameter benchmark examples with sketches of the stability proofs. The designs 
are then extended to to reaction—advection-diffusion plants in 2D and plants with spatially varying 
(functional) parametric uncertainties. The chapter ends with the output-feedback adaptive design for 
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reaction—advection-diffusion systems with only boundary sensing and actuation. These systems have an 
infinite relative degree, infinitely many unknown parameters and are open-loop unstable, representing 
the ultimate challenge in adaptive control for PDEs. The detailed proofs for the designs presented here 
are given in [13,23-26]. 


71.10 Lyapunov Design 


Consider the plant 
ur(x, tf) = uxx(x, t) + Au(x, ft), O<x <1, (71.116) 
u(0, t) = 0, (71.117) 
ux(1, t) = U(t), (71.118) 


where i is an unknown constant parameter. We use the Neumann boundary controller (Equation 71.21) 
with k(x, y) given by Equation 71.15 and } replaced by its estimate i: 


h (/ ve 2) 


1-8 


+ 1 
U(t)= Put) ifs u(&, t)dé. (71.119) 
0 


Straightforward calculation shows that invertible change of variables 


i; (/ L(x? 2) 


x 
w(x, t) = u(x, n-|[ k(x, —)u(E, t) dé, k(x, &)= RE (71.120) 
: Na? = §?) 
maps Equations 71.116 through 71.119 into 
a [*E = 
wyz(x, t) = Wyx(x, £) + if 5 t) dé + w(x, t), (71.121) 
0 
w(0, t) = w,(1, t) = 0, (71.122) 
where ) = i — i is the parameter estimation error. 
Consider the Lyapunov function* 
V= * tog (1+ IIwll2) + 72 (71.123) 
5 ie : 
Its time derivative along the solutions of Equations 71.121 and 71.122 is 
2 sl x ; 2 
nA Jy wx w(&)dé)dx  - 1: 
pa Jo wle) fo (648) i( pee ) (71.124) 
1+ wll? 2 1+ |lw|l? y 1 +llwil? 


(the calculation involves integration by parts). Based on Equation 71.124, we choose the update law 


jeer lisa O<y<lL. (71.125) 
1+ |lwll? 
Weare going to show that with this update law, u(x, t) is regulated to zero for all x € [0, 1], for arbitrarily 
large initial data u(x, 0) and for an arbitrarily poor initial estimate (0). 


* In Sections 71.10 through 71.15, we suppress time dependence to simplify the notation, that is, w(0) = w(0, £), || wl] = 


\|wG, OI], ete. 
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Note that V depends only on time and contains the logarithm of the spatial Lz norm [19]. This results in 
the normalized update law (Equation 71.125) and slows down the adaptation, which is necessary because 
the control law (Equation 71.119) is of certainty equivalence type. An additional measure of preventing 
overly fast adaptation in Equation 71.125 is the restriction on the adaptation gain (y < 1). 

Using Cauchy and Poincare inequalities, one obtains 

< Falwell (71.126) 


1 x 
| [ wile) ( [ zw(@)de) ds a 


Substituting Equations 71.126 and 71.125 into Equation 71.124 and using the fact that |h| < y (see 


Equation 71.125), we obtain 
. Y Il wx I 
V<-(1 : (71.127) 
( V3 ) 1+ |lw|l? 
This implies that V(t) remains bounded for all time whenever 0 < y < /3. From the definition of V it 
follows that ||w|| and } remain bounded for all time. To show that w(x, t) is bounded for all time and for 
all x, we estimate (using Agmon, Young, and Poincare inequalities): 


A 


= nh 
il wel? = —[Wexll? + Mllwsll? + 7 (wD)? = Ihe’) 


< —(1—y)||weell? + Mlwell? < Alle ll?. (71.128) 


Integrating the last inequality, we obtain 


t 
IIw(t)|I? < llwx(O)I* +2 sup |d(x)| , \|wx(t) ||? dt. (71.129) 


0<t<t 


Using Equation 71.127 and the fact that ||w|| is bounded, we obtain 


2 
[iwcorasaro fi OT dt <0, (71.130) 


where C is the bound on ||w||?. From Equations 71.129 and 71.130, we get that || wx.||? is bounded. Since 
maxxey0,1] |w(x)|? < ||wx||?, we observe that w(x, t) is bounded for all x and f. 


Next, we prove regulation of w(x, f) to zero. Using Equations 71.121 and 71.122, it is easy to show that 


2 2 . Y 2 
< Iw ll + G ae -) |wIl". (71.131) 
Since ||w|| and ||w,|| have been proven bounded, it follows that £ \wIil? is bounded, and thus || w(t)|| 
is uniformly continuous. From Equation 71.130 and Poincare inequality, we get that ||w||? is integrable 
in time over the infinite time interval. By Barbalat’s lemma, it follows that ||w|| > 0 as t > oo. The 
regulation in the maximum norm follows from Agmon’s inequality. 

Having proved the boundedness and regulation of w, we now set out to establish the same for u. We 
start by noting that the inverse transformation to Equation 71.120 is [21] 


hh ( K(x? — 2) 


RG =) 


u(x) = w(x) + [icoemeyas, I(x, &) = —hE (71.132) 
0 


Since i is bounded, the function I(x, &) has bounds L) = maxo<e<x<1 I(x, £)?,L,= maxo<t<x<1 i (x, &). 
It is straightforward to show that 


IIull2 <2 (145? +412) IIwell2, (71.133) 
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Noting that u(x, t)? < ||u,||? for all (x, t) € [0, 1] x [0, 00) and using the fact that || w,|| is bounded, we get 
uniform boundedness of u. To prove regulation of u, we estimate from Equation 71.132 


\Jull? < 211+ L,)||wIl?. (71.134) 


Since ||w]| is regulated to zero, so is ||u|]. By Agmon’s inequality u(x, t)* < 2||u||||ux|], where ||u,|| is 
bounded. Therefore u(x, ft) is regulated to zero for all x € [0, 1]. 

The Lyapunov design incorporates all the states of the closed-loop system into a single Lyapunov 
function and therefore Lyapunov adaptive controllers possess the best transient performance properties. 
However, this method is not applicable as broadly as the certainty equivalence approaches, which we 
consider next. 


71.11 Certainty Equivalence Design with Passive Identifier 


Consider the plant 
uz(x, t) = Uxx(x, f) + Ku(x, t), (71.135) 
u(0, t) = 0, (71.136) 
u(1, t) = U(E), (71.137) 


where a constant parameter ) is unknown. We use the Dirichlet controller (Equation 71.16) with \ 


replaced by its estimate i: 
_ 1 (yia-2)) 
U(t)=—h / € 
° yk 8?) 


Following the certainty equivalence principle, first we need to design an identifier which will provide the 
estimate i. 
Consider the following system: 


u(&, t)dé. (71.138) 


1 
ity = thy thuty?(u— i) [ u?(x) dx, (71.139) 

0 
iu(0) = 0, (71.140) 
u(1) = u(1). (71.141) 


Such a system is often called an “observer,” even though it is not used here for state estimation (the entire 
state u is available for measurement in our problem). The purpose of this “observer” is to help identify 
the unknown parameter. This identifier employs a copy of the PDE plant and an additional nonlinear 
term. We will refer to the systems (Equations 71.139 through 71.141) as a “passive identifier.” The term 
“passive identifier” comes from the fact that an operator from the parameter estimation error } = } — i 
to the inner product of u with u — @ is strictly passive. The additional term in Equation 71.139 acts as 
nonlinear damping whose task is to slow down the adaptation. 

Let us introduce the error signal e = u — u. Using Equations 71.135 and 71.136 and Equations 71.139 
through 71.141, we obtain 


ep = xy + hu — yell ull’, (71.142) 
e(0) = e(1) =0. (71.143) 
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With the Lyapunov function 


ae ae Ge 
V==] e(x)dx+—, (71.144) 
2 0 2y 
we obtain : 
7 2 2 2,2 a ¥ : nA 
V=—llexl—yollell yall + f elx)u(x) d are (71.145) 
0 
Choosing the update law 
; 1 
hey | (u(x) — it(x))u(x) dx, (71.146) 
0 
we obtain 
V =—llexll? — y*llell” lull’, (71.147) 


which implies V(t) < V(0) and from the definition of V we get that i and |lel| are bounded functions of 
time. Integrating Equation 71.147 with respect to time from zero to infinity we get that the spatial norms 
||ex|| and |le]| |||] are square integrable over infinite time. From the update law (Equation 71.146) we have 


|h| < yllell||ul|, which shows that i is also square integrable in time. 
One can show that the transformation 


(x) = Ux) — / k(x, u(y) dy (71.148) 
0 
with k given by Equation 71.120, maps Equations 71.139 through 71.141 into the following “target system” 
fy ting h [O Siie) dé + + Yuen (71.149) 
0 


W(0) = W(1) = 0, (71.150) 


where e is the transformed estimation error 
e1(x) = e(x) — / k(x, y)e(y) dy. (71.151) 
0 


We observe that in comparison to nonadaptive target system (plain heat equation) two additional terms 


appeared in Equation 71.149, one is proportional to i and the other to the estimation error e. The 
identifier guarantees that both of these terms are square integrable in time. 

Since 4. is bounded, and the functions k(x, y) and I(x, y) are twice continuously differentiable with 
respect to x and y, there exist constants M,, M2, and M3 such that 


lel <= Millell, [lull < Wall + llell < Mallwl| + llell (71.152) 
I[uxl] < [tel] + lexll < M3 llwell + llexll- (71.153) 


Using Equation 71.152, Young’s, Cauchy-Schwarz, and Poincare inequalities along with the identifier 
properties, one can obtain the following estimate: 


ld, 
sal 


1 
2<— |W +h |]? 4h, 71.154 
rer Ir < 7a + hw +h ( ) 


where 1, Jy are some integrable functions of time on [0, 00). Using Lemma B.6 from [12] we get bounded- 
ness and square integrability of || v||. From Equation 71.152 we get boundedness and square integrability 


of ||7||, ||u|], which also proves boundedness of h. 


Boundary Control of PDEs 71-17 


In order to get pointwise in x boundedness, one estimates 


ea pss 1 242 
-— |] Wdx<-— =|", |? + —— + Ooty’ lull??? Mi llell, 71.155 
buble ene 3! Pal r Qro + yell“)! [lel ( ) 
i oa ee | a es (ee ee 
—— e. dx < ——=llex||“ + —|A]*||ul|*. 71.156 
OTe Pea ae FL ll A “|[ull ( ) 


Since the right-hand sides of Equations 71.155 and 71.156 are integrable, using Lemma B.6 from [12] 
we get boundedness and square integrability of ||W,|[, |lex|], and ||@«|[, ||ux|| (from Equation 71.153). 
By Agmon’s inequality, we get that # and u are bounded for all x € [0, 1]. 

To show the regulation of u to zero, note that 


< |Xillell lull < 00. (71.157) 


ld 2 
Salt 


The boundedness of |(d/dt)||w||?| follows from Equation 71.154. By Barbalat’s lemma, we obtain 
||| — 0, |lel]| > 0 as t > oo. Since the transformation (Equation 71.148) is invertible, we have ||z|| > 0 
and from Equation 71.152 ||u|| + 0. Using Agmon’s inequality and the fact that ||1,|| is bounded, we 
obtain 


lim max |u(x, t)| = 0. (71.158) 


too xE[0,1] 


71.12 Certainty Equivalence Design with Swapping Identifier 


The certainty equivalence design with swapping identifer is perhaps the most common method of param- 

eter estimation in adaptive control. Filters of the “regressor” and of the measured part of the plant are 

implemented to convert a dynamic parameterization of the problem (given by the plant’s dynamic model) 

into a static parametrization where standard gradient and least-squares estimation techniques can be used. 
Consider the plant 


ut(x, ft) = Ux (x, t) + Ku(x, £), (71.159) 
u(0,t) = 0, (71.160) 
u(1, t) = U(t) (71.161) 


with unknown constant parameter i. We start by employing two filters: the state filter 


V¢(x, t) = Vax(x, £) + u(x, t), (71.162) 
v(0, t) = v(1, t) = 0, (71.163) 
and the input filter 
ne(x, t) = Nex(x, t), (71.164) 
n(0, t) = 0, (71.165) 
n(1, t) = U(t). (71.166) 
The “estimation” error 
e=u—rv—-y (71.167) 
is then exponentially stable: 
ez (x, t) = Cxx(X, t), (71.168) 


e(0,t) = e(1,t) = 0. (71.169) 
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Using the static relationship (Equation 71.167) as a parametric model, we implement a “prediction error” 
as 


@é=u—-ihv—-n, @=et+h. (71.170) 


We choose the gradient update law with normalization 


s fy ex)v(x) dx 


oye (71.171) 
T+ IP 
With the Lyapunov function 
ae l 
V= al e dx+—i (71.172) 
2 Jo 8y 
one obtains “ 
Vv <—)pegl2 Nel? (71.173) 
2 81+ ]|v|I? 
This gives the following properties: 
é ay 
Hell , X% are bounded, (71.174) 
1+ |Iv|/? 
\lell ‘ ; ote 
———., are square integrable in time over [0, oo). (71.175) 
1+ |vll? 


In contrast with the passive identifier, the normalization in the swapping identifier is employed in the 


update law. This makes } not only square integrable but also bounded. 
We use the controller (Equation 71.138) with the state u replaced by its estimate kv + n: 


h (V fa ”) 


1 
U(t)= af E . (Av(E, t) + n(&, £)) dé. (71.176) 
. AC — §2) 
The transformation (Equation 71.148) is also modified to use the estimate of the state: 
W(x) = hv(x) + n(x) — / k(x, &)(Av(&) + 0(8)) dé, (71.177) 
0 


where k(x, £) is given by Equation 71.120. Using Equations 71.162 through 71.166 and the inverse trans- 
formation 


Rv(x) + n(x) = (x) + [ bc &)i(E) dé (71.178) 
with I(x, £) given by Equation 71.132, one can obtain the following PDE for w: 
ft = Wee thV+ A [Ge — k(x, one)) de +h (e = [ k(x, 8)0(6) as) (71.179) 
W(0) = W(1) = 0. (71.180) 
Rewriting the filters (Equations 71.162 through 71.163) as 
reac Ot [Thar E) a, (71.181) 
v(0) = v(1) = 0, ; (71.182) 


we obtain two interconnected systems for v and w, Equations 71.179 through 71.182, which are driven 


by the signals }. and é with properties (Equations 71.174 and 71.175). Note that the situation here is more 
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complicated than in the passive design where we had to analyze only the w-systems (Equations 71.149 
through 71.150). While the signal v in Equations 71.179 through 71.180 is multiplied by a square- 
integrable signal i, the signal w in the v-systems (Equations 71.181 through 71.182) is multiplied by a 
bounded but possibly large gain 1. Therefore, to show boundedness of ||| and ||v||, we use a weighted 
Lyapunov function 

WHA? + |v, (71.183) 


where A is a large enough constant. One can then show that 
. 1 
W <-—W+BhW, 71.184 
aa 3 ( ) 


where /3(f) is an integrable function of time over [0, 00), and with the help of Lemma B.6 from [12] we 
get the boundedness of ||w|| and ||v||. Using this result it can be shown that 


Os A 
a (tell? + Ilvell’) < [| Weel]? = [!Vxccll? + la (71.185) 


with integrable /4(¢), which proves that || w, || and ||v,|| are bounded. From Agmon’s inequality we get that 
w and v are bounded pointwise in x. By Barbalat’s lemma, || w|| — 0, ||v|| > 0.as t — oo. From Equations 
71.178 and 71.167 we get the pointwise boundedness of ny and u and ||u|| — 0. Finally, the pointwise 
regulation of u to zero follows from Agmon’s inequality. 

The swapping method uses the highest order of dynamics of all identifier approaches. Lyapunov 
design has the lowest dynamic order as it only incorporates the dynamics of the parameter update, 
and the passivity-based method is better than the swapping method because it uses only one filter, as 
opposed to “one-filter-per-unknown-parameter” in the case of the swapping approach. Despite its high 


dynamic order, the swapping approach is popular because it is the most transparent (its stability proof is 
the simplest due to the static parametrization) and it is the only method that incorporates least-squares 
estimation. 


71.13 Extension to Reaction—Advection—Diffusion Systems 
in Higher Dimensions 


All the approaches presented in Sections 71.10 through 71.12 can be readily extended to reaction- 
advection-diffusion plants and higher dimensions (2D and 3D). As an illustration, consider a 2D plant 
with four unknown parameters ¢, b1, bz, and i: 


Up = E(Uxx + Uyy) + Di Uy + boty + du (71.186) 


on the rectangle 0 < x < 1,0 < y < Lwith actuation applied on the side with x = 1 and Dirichlet boundary 
conditions on the other three sides. 
We choose to design the scheme with passive identifier. We introduce the following “observer” 


ily = 8(Giaxe + flyy) + by dy + bofin + hut Y(u—a)||Vull?, (71.187) 
a=0, (x,y) € {[0, 1] x [0,1]}\{x = 1}, (71.188) 
w=u, x=1, O<yK<1. (71.189) 


There are two main differences compared to 1D case with one parameter considered in Section 71.11. 
First, since the unknown diffusion coefficient ¢ is positive, we must use projection to ensure € > ¢ > 0: 
0 €=e and 1t<0, 


(71.190) 
t else. 


Proj,{t} = 


Although this operator is discontinuous, it can be easily modified to avoid dealing with Filippov solutions 
and noise due to frequent switching of the update law; see [13] for more details. However, we use Equation 
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71.190 here for notational clarity. Note that € does not require the projection from above and all other 
parameters do not require projection at all. 

Second, we can see in Equation 71.187 that while the diffusion and advection coefficients multiply the 
operators of ii, the reaction coefficient multiplies u in the observer. This is necessary in order to eliminate 
any \-dependence in the error system so that it is stable. 

The update laws are 


, 1 pl 
& = —yoProj, {f / Ux (Ux — Ux) + Uy(uy — ty) dx ay| F (71.191) 
a 0 0 
a 1 1 A 1 1 
n= [ / (u— it)ux dx dy, in=v [ / (u— t)uy dx dy, (71.192) 
0 0 0 0 
t di ab 
= » | / (u—u)u dx dy, (71.193) 
0 0 
and the controller is 
. 
‘ q =(1-&?) 
TX _ ba-8) i : 
u(1,y, t) = -| Zoe we ——_________*_ii(&, y, t) dé. (71.194) 
i A 
h 
—(1='§7) 
E 


The results of the simulation of the above scheme are presented in Figures 71.1 and 71.2. The true 
parameters are set to ¢ = 1,b; = 1, b2 = 2, = 22, and L = 2. With this choice the open-loop plant has 
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FIGURE 71.1 The closed-loop state for the plant (Equation 71.186) with adaptive controller (Equation 71.194) and 
passive identifiers (Equations 71.187 through 71.193) at different times. 
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FIGURE 71.2 The parameter estimates for the plant (Equation 71.186). 


two unstable eigenvalues at 8.4 and 1. All estimates come close to the true values at approximately t = 0.5 
and after that the controller stabilizes the system. 


71.14 Plants with Spatially Varying Uncertainties 


The designs presented in Sections 71.10 through 71.12 can be extended to plants with spatially varying 
unknown parameters. For example, for the plant 


Uz (x, t) = Ux (x, t) + A(x)u(x, £), (71.195) 
ux(0, t) = 0, (71.196) 


the Lyapunov adaptive controller would be 


1 
u(1, t) = k(1, 1)u(1, t) +} ky (1, &)u(&, t) dé (71.197) 


0 


with 


u(x,t) (wont) — J RE, x)w(E, a8) 
1+ TWOP 


> 


he(x, th=y 
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where i(t,x) is the online functional estimate of A(x), w(x) = u(x) — ah k(x, &)u(&)dé, and the kernel 
K(x, s)= ken(X, &) is obtained recursively from 


Z fo 
lotuy <-> [i d (¢) de, (71.198) 
7 
into) =hioy+ fe f i (6-0) ki (¢+0,¢—0) do dt, i=0,1,...,n, (71.199) 
xs JO 
2 


for each new update of i(x, ft). Stability is guaranteed for sufficiently small y and sufficiently high n. 
The recursion (Equation 71.199) was proved convergent in [21]. The certainty equivalence designs with 
passive and swapping identifiers can also be extended to the case of functional unknown parameters using 
the same recursive procedure. For further details, the reader is referred to [23]. 


71.15 Output-Feedback Design 


Consider the plant 
ur (Xx, t) = Uxx(x, t) + A(x)ulx, 1), (71.200) 
ux(0, t) = 0, (71.201) 
u(1,t) = U(t), (71.202) 


where (x) is an unknown continuous function and only the boundary value u(0, t) is measured. 
The key step in the output-feedback design is the transformation of the original plant (Equations 71.200 
through 71.202) into a system in which unknown parameters multiply the measured output. 


71.15.1 Transformation to Observer Canonical Form 


One can show that the transformation 
v(x, t) = u(x, t) — / p(x, y)u(y, t) dy, (71.203) 
0 


where p(x, y) is a solution of the PDE 


Pxx(X5¥) — Py y) = (yp y), (71.204) 
ply) =0, (71.205) 

1 
p(x, x) = ; / X(s) ds, (71.206) 


maps the systems (Equations 71.200 through 71.202) into 


Vz(x, t) = Vex(x, t) + O(x)v(0, t), (71.207) 
vx(0, t) = 91 v(0, £), (71.208) 
v(1, t) = U(t), (71.209) 
where 
O(x) = —py(x,0), 6 = —p(0, 0) (71.210) 


are the new unknown functional parameters. 
The systems (Equations 71.207 through 71.209) is the PDE analog of observer canonical form. 
Note from Equation 71.203 that v(0) = u(0) and therefore v(0) is measured. The transformation 
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(Equation 71.203) is invertible so that stability of v implies stability of u. Therefore, it is enough to 
design the stabilizing controller for y-system and then use the condition u(1) = v(1) (which follows from 
Equation 71.205) to obtain the controller for the original system. The new unknown parameters 0(x) 
and 0) are going to be estimated directly, therefore there is no need to solve the PDE (Equations 71.204 
through 71.206) for the control scheme implementation. 


71.15.2 Filters 


The unknown parameters @ and 6(x) enter the boundary condition and the domain of the v-system. 
Therefore, the following output filters are needed: 


1(x, t) = Pxx(x, t), (71.211) 
x(0, t) = u(0, t), (71.212) 
o(1, t) =0, (71.213) 
and 
,(x, t,&) = Pyx(x, t, &) + 8(x — €)u(0, £), (71.214) 
,(0,f,£) = (1, t,£) =0. (71.215) 


Here the filter 0 = O(x, t, &) is parametrized by € € [0, 1] and 8(x — €) is the delta function. The reason 
for this parametrization is the presence of the functional parameter (x) in the domain. Therefore, loosely 
speaking, we need an infinite “array” of filters, one for each x € [0, 1] (since the swapping design normally 
requires one filter per unknown parameter). We also introduce the input filter 


Wil, t) = Vax(x, t), (71.216) 
Wx(0, t) = 0, (71.217) 
Wl, t) = U(d). (71.218) 


It is straightforward to show now that the error 


1 
e(x, t) = v(x, t) — W(x, t) — 91. (x, t) — / O(E) B(x, t, &) dé (71.219) 
0 
satisfies the exponentially stable PDE 
e(x, t) = exx(x, t), (71.220) 
2,(0,t) = e(1,t) = 0. (71.221) 


Typically, the swapping method requires one filter per unknown parameter and since we have functional 

parameters, infinitely many filters are needed. However, it is possible to reduce their number down to 

only two by representing the state P(x, &, t) algebraically through (x,t) at each instant t. Comparing 

the explicit solutions of the systems (Equations 71.211 through 71.213) and (Equations 71.214 through 
71.215), one can show that ® can be represented as ® = F + AF, where AF satisfies 

AF, = AF yx, (71.222) 

AF, (0, £, t) = AF(1, £,t) = 0, (71.223) 


and F(x, &, t) is obtained from the filter @ through the solution of the PDE 


Fyx(x, §) = Fee (x, §), (71.224) 
F(0,§) = —o(&), (71.225) 
F,(0,&) = Fe(x, 0) = F(x, 1) =0. (71.226) 
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It is then easy to show that the signal 


1 
e(s) = v(x) — la) — 816(8) — | 0(G)Fla 8) de (71.227) 
0 
is governed by the exponentially stable heat equation: 
€; (x, t) = exx(x, t), (71.228) 
e,(0, t) = e(1, t) = 0. (71.229) 


Therefore, instead of solving the infinite “array” of parabolic equations 71.214 through 71.215, one can 
solve only two dynamic equations for filters @ and y and compute the solution of the standard wave 
equation 71.224 through 71.226 at each time step. 


71.15.3 Update Laws 
The following equation serves as a parametric model: 
1 
e(0) = v(0) — (0) — 01(0) +f 9(5)b(§) dé. (71.230) 
0 


The estimation error is ; 
(0) = (0) — W(0) —b,9(0)+ f H(E)0CE) a. (71.231) 
0 
We employ the gradient update laws with normalization 


e(0) (x) F e(0) (0) 


6,(x, t) = —y(x) , = ; (71.232) 
MNT +O +20) 11+ 12 + 420) 
where y(x) and y; are positive adaptation gains. 
With the Lyapunov function 
1 Vig 1 §2(x) 
V = -|lell?7 + —@ +f dx (71.233) 
2 avi Jo 2y(x) 
one obtains ‘ 
: 1 1 e-(0 
V <—=llexll? ats : (71.234) 
2 21+ |Io||? + $7(0) 
which gives the following properties of the adaptive laws: 
e(0 Bet. yee Vesa 
a ,[[6r[1, 91, [|9l], 81 are bounded, (71.235) 
1+ Ill? + 67(0) 
e(0) ay A ee 
, ||9,||, 91 are square integrable in time. (71.236) 
1+ [Ip]? + 67(0) 
71.15.4 Controller 
The PDE for the estimated gain kernel for the plant (Equations 71.207 through 71.209) is 
Karel, y) = kyy(x,y)s (71.237) 
ky (x, 0) = 61k (x, 0) + A(x) — ip k(x, y)6(y) dy, (71.238) 
0 


k(x, x) = 61. (71.239) 
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The solution to this PDE is k(x, y) = k(x — y), where k satisfies the simple integral equation 


ca) =0,- [Hy ay | fa f°” @(s) as "Ore (71.240) 
0 0 0 


which has to be solved online. 
The controller is given by 


1 7 1 ‘ 
Ue K(1—y) (wor+o0r+ i Fy, 06) a8) dy. (71.241) 


Using the transformation 
1 
w(x) = Ws) +6106) + [RG 89816) a 
0 


x 1 
-| K(x — y) jvon+boon+ [ F806) a8 dy (71.242) 


and analyzing stability properties of the interconnected PDEs for w and 6 similarly to the way it is done 
in Section 71.12, one can show that all the signals in the closed-loop system are bounded and u(x, ft) is 
regulated to zero uniformly in x € [0, 1]. 

The design methodology presented above can also be applied to general reaction—-advection-diffusion 
systems 


Up = €(X)Uxx + b(x)ux + A(x)u+ g(x)u(0) + [fo.nun dy (71.243) 

0 
ux(0) = —qu(0), (71.244) 
u(1) =U, (71.245) 


where (x) is known and b(x), d(x), g(x), f(x, y), and q are unknown parameters. 


Further Reading 


Due to space constraints, the material presented in this chapter does not include all the backstepping- 
based designs developed up to the time of writing this chapter. Most of those can be accessed in two 
recently published books, [15,28]. 

In particular, the stabilizing controllers in Sections 71.2 through 71.4 can be modified so that a certain 
meaningful cost functional is minimized, providing stability margins [21] (the so-called inverse-optimal 
design). 

In addition to the design for the shear beam in Section 71.8, which is presented in more details in [16], 
backstepping controllers have also been developed for the Euler-Bernoulli beam [20]. 

Designs for nonlinear parabolic PDEs are presented in [29,30]. 

Finally, the backstepping method is particularly suited for dealing with PDE-ODE or PDE-PDE 
cascades, for example, systems with actuator/sensor dynamics in the form of the pure delay [14]. 
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72.1 Introduction 


Recent years have been marked by dramatic advances in the field of active flow control. This development 
can be credited not only to advances in the various fields that intersect through this discipline (such as 
control theory, fluid mechanics, PDE theory, and numerical methods), but also to technological develop- 
ments such as Micro-Electro-Mechanical Systems (MEMS) sensors and actuators and the ever-increasing 
prowess of last-generation computers, that have augmented the possibilities of effective implementation 
in both real-life and numerical experiments. However, the area is far from being mature with still many 
opportunities and challenging open problems. 

Efforts over the last few years have led to a wide range of developments in many different directions, 
reflecting the interdisciplinary character of the research community. However, most implementable devel- 
opments so far have been obtained using discretized versions of the plant models and finite-dimensional 
control techniques. In contrast, in this chapter we present a design method that is based on the “contin- 
uum” version of the backstepping approach, applied to the partial differential equation (PDE) model of 
the flow. The postponement of the spatial discretization until the implementation stage offers advantages 
that range from numerical to analytical. In fact, this design method offers a unique physical intuition by 
forcing the closed-loop system to dynamically behave as a set of well-damped heat equation PDEs. 

In the next sections we present an example of feedback control and observer design for an incompress- 
ible 3D magnetohydrodynamic (MHD) channel flow, also known as the Hartmann flow, a benchmark 
model for applications such as cooling systems (computer systems, fusion reactors), hypersonic flight, and 
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propulsion. In this flow, an electrically conducting fluid moves between parallel plates and is affected by an 
imposed transverse magnetic field. The velocity and electromagnetic fields are mathematically described 
by the MHD equations, which are the Navier-Stokes equation coupled with the Maxwell equations. For 
zero magnetic field or nonconducting fluids, the plant reduces to the 3D Navier-Stokes channel flow, and 
hence, the control and observer designs apply to the 3D Navier-Stokes system when the magnetic field 
parameter is set to zero. 

Owing to the high complexity of the Navier-Stokes problem (further aggravated by the presence of 
Maxwell’s equations), most derivations and proofs are skipped or at best only sketched. Further details 
and methods for flow control can be found in the recent book by Vazquez and Krstic [26]. The main 
method used in the designs is the backstepping method for PDEs. Its basics are outlined in Chapter 71 
on “Boundary control for PDEs” by Krstic and Smyshlyaev in this Handbook. Some very elementary 
background on Lyapunov stability (see Chapter 43 by Khalil in this Handbook) and basic PDE theory is 
assumed in this chapter, but no previous familiarity with flow control is necessary. 

In Section 72.2, we present the mathematical model of the Hartmann flow, and then study its equilib- 
rium profile. We linearize the system around the equilibrium and derive the plant equations in Fourier 
space. We follow Section 72.3, where we design (in Fourier space) a feedback control law that stabilizes 
the linearized system using the backstepping method. In Section 72.4 we study the problem of designing 
a nonlinear estimator, with linear output injection, using the backstepping observer design method. The 
observer and controller, combined together, provide a stabilizing output feedback control law. We finish 
with a brief literature review in Section 72.5. 


72.2 Channel with MHD Flow 


We consider a benchmark 3D MHD channel flow, known as the Hartmann flow. This flow consists of an 
incompressible conducting fluid enclosed between two parallel plates, separated by a distance Lp, under 
the influence of a pressure gradient VP parallel to the walls and a magnetic field By normal to the walls, 
as shown in Figure 72.1. Under the assumption of a very small magnetic Reynolds number 


Rem = WUmoUcLp <1, (72.1) 


where | is the magnetic permeability of the fluid, o the conductivity of the fluid, and U;, the reference 
velocity (maximum velocity of the Hartmann equilibrium profile), the dynamics of the magnetic field can 
be neglected and the dimensionless velocity and electric potential field are governed by the inductionless 
MHD equations [17]. 

We set nondimensional coordinates (x,y,z), where x is the streamwise direction (parallel to the 
pressure gradient), y the wall normal direction (parallel to the magnetic field), and z the spanwise 


FIGURE 72.1 Hartmann flow. 
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direction, so that (x, y, z) € (—00, 00) x [0, 1] x (—00, 00). The governing equations are 


AU 

U; = a UU, — VUy — WU, — Px +Noz — NU, (72.2) 
AV 

Varig, = Ue Wy We (72.3) 
AW 

W= Re UW; — VW, — WW, — P, —Nox —NW, (72.4) 

Ao = Uz — Wx; (72.5) 


where U,V, and W denote, respectively, the streamwise, wall-normal and spanwise velocities, P the 
pressure, ¢ the electric potential, Re = (U-Lp)/v is the Reynolds number, N = (oLpBS) /(pU-) the Stuart 
number, and A = Oyy + Oy + Ozz denotes the standard Laplacian operator. The notations U; and U, 
(and other subscripts such as y, z, xx, etc.) denote the partial derivatives of U with respect to ¢ and x, 
respectively. Since the fluid is incompressible, the continuity equation is verified 


Ux + Vy + Wz =0. (72.6) 


The boundary conditions for the velocity field are 


U(t, x, 0, z) = U(t,x, 1,z) = U-(t,x, z); (72.7) 
V(t, x, 0, z) = V(t,x,1,z) = V-(t, x, z), (72.8) 
W(t, x,0,z) = W(t, x, 1,z) = W-(t,x,z); (72.9) 


where U;(t,x,Z), Vc(t,x,z), and W,(t,x,z) denote, respectively, the actuators for streamwise, wall- 
normal, and spanwise velocity in the upper wall. Assuming perfectly conducting walls, the electric 
potential must verify 


o(t, x,0,z) = 0, (t,x, 1,z) = ®-(t, x, z), (72.10) 


where ®,(t, x, Z) is the imposed potential (electromagnetic actuation) in the upper wall. 
We assume that all actuators can be independently actuated for every (x, z) € R*. Note that no actuation 
is done inside the channel or at the bottom wall. 


Remark 72.1 


If we set N = 0 (zero magnetic field, or nonconducting fluid) in Equations 72.2 through 72.5, they reduce 
to the classical Navier-Stokes Equations without body forces. Then Equations 72.2 through 72.4, 72.6, 
and Equations 72.7 through 72.9 describe a pressure-driven channel flow, known as the Poiseuille flow. 


72.2.1 Hartmann Equilibrium Profile 


The equilibrium profile for systems (Equations 72.2 through 72.5) with no control can be found explicitly 
(as is also the case with the Poiseuille solution for Navier-Stokes channel flow). We assume a steady 
solution with only one nonzero nondimensional velocity component, U®, that depends only on the y 
coordinate. Substituting U* in Equation 72.2, one finds that it verifies the following equation: 


a) 


— P? — NU“(y), (72.11) 
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whose nondimensional solution is, setting P° such that the maximum velocity (centerline velocity) is 
unity, 


sinh(H(1 — y)) — sinh H + sinh(Hy) 


uO) = 2 sinh H/2 — sinh H , Wer) 
Vo=W=6° =0, (72.13) 
pee ey (72.14) 

2 sinh H/2— sinh H 
f =; = 0.77 = U*(y). (72.15) 


where H = /ReN = BoLy, | or is the Hartmann number. In Figure 72.2 (left) we show U*(y) for different 
values of H. Since the equilibrium profile is nondimensional the centerline velocity is always 1. For H = 0, 
the Poiseuille equilibrium profile is obtained, which is parabolic. In Figure 72.2 (right) we show Uy(y), 


proportional to shear stress, whose maximum is reached at the boundaries and grows with H. 


72.2.2 The Plant in Wave Number Space 
Define the fluctuation variables 


u(t, x, y) = U(t,x, y) — U*(y), (72.16) 
p(t, x,y) = P(t, x, y) — P*(y), (72.17) 
where U%(y) and P*(y) are, respectively, the equilibrium velocity and pressure given in Equations 72.12 


and 72.14. The linearization of Equations 72.2 through 72.4 around the Hartmann equilibrium profile, 
written in the fluctuation variables (u, V, W, p, o), is 


Au e . 
lr = By — Ux — UZ Q)V — pe + Noe — Nu, 218) 
AV oy 
Ve = pee (y) Vx = Py» (72.19) 
AW 
Wi = S~ — UW) We — pz — NOx — NW. (72.20) 
e 


(b) | 


(a) 1 a 
ta ta ~ 
0.5 + 0.5 + 
0 0 a 
50 50 


FIGURE 72.2 Streamwise equilibrium velocity U°(y) (left) and Uy (y) (right), for different values of H. Solid, H = 0; 
dash-dotted, H = 10; dashed, H = 50. 
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The equation for the potential is 
Ad = uz — Wz, (72.21) 


and the fluctuation velocity field verifies the continuity equation, 


Ux + Vy + W, =0, (72.22) 
and the following boundary conditions: 
u(t, x, 0, z) = W(t, x, 0, z) = V(t, x, 0,z) = 0, (72.23) 
u(t, x, 1,z) = U-(t, x, Z), (72.24) 
V(t, x, 1,z) = V-(t, x, Z), (72.25) 
W(t, x, 1,z) = W-(t,x,Z), (72.26) 
(t,x, 0,z) =0, o(t, x, 1,z) = ®,(t, x, z). (72.27) 


Since the plant is linear and invariant to shifts in the streamwise (x) and spanwise (z) directions, we 
use a Fourier transform in x and z coordinates. The transform pair (direct and inverse transform) is 
defined as 


[o) CO 
flkxs Ys kz) = / / flxyrz)e2milkexthe2) dz dx, (72.28) 
— OO. —CoO 


CO [o) 
f@.92)-= / / f (keryr keyermtilhes thee) dk, dk. (72.29) 
=o. Oo 


Note that we use the same symbol f for both the original f(x, y,z) and the transform f (kx, y, kz). The 
quantities k, and k, are referred to as the “wave numbers.” 
Denoting a? = 4n(k2 + k2) and B = 2nik,U*%, the plant equations in Fourier space are 


uy = Sti Bu — UfV — Ink, ip + 2nk,iNd — N (72.30) 
t= Re u—U, hyip + 20kziNb — Nu, . 
—a’V+ Vyy 
Vie BV —py, (72.31) 
2 
ay BW —2xk,ip — 20k,iNo — NW (72.32) 
— Re wtk,ip ttk,iNo ; : 


The continuity equation in wave number space is expressed as 


2nik,u+ Vy +210k,W =0, (72.33) 
and the equation for the potential is 
—a7 + byy = 2mi (kzu—k,W). (72.34) 
The boundary conditions are 
u(t, kx, 0, kz) = W(t, ky, 0, kz) = V(t, ky, 0, kz) = O(t, ky, 0; kz) = 0, (72.35) 
u(t, ky, 1, kz) = Uclts kes kz), W(ts keys 1 kz) = Velts keys kz), (72.36) 


W(t, kx, 1, kz) = Welt, kxs kz), b(t, kx, 1, kz) = Oct, ky, kz). (72.37) 
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72.3 Boundary Control Design 


The Hartmann and Poiseuille flows are unstable for large Reynolds numbers. To guarantee stability, our 
design task is to design feedback laws U,, V-; W-, and ®,, so that the origin of the velocity fluctuation 
system is exponentially stable. Full state knowledge is assumed in this section. 

We design the controller in wave number space. Note that Equations 72.30 through 72.37 are uncoupled 
for each wave number. Therefore, the set of wave numbers k2 + k? < M? (for some large M > 0), which we 
refer to as the controlled wave number range, and the range k2 + k? > M?, the uncontrolled wave number 
range, can be studied separately. If stability for all wave numbers is established, stability in physical space 
follows. The number M, which will be computed in Section 72.3.2, is a parameter that ensures stability 
for uncontrolled wave numbers. 

We define x, a truncating function, as 


X(kxs kz) = | i iG isa (72.38) 
Then, we reflect that we only use control for small (controlled) wave numbers by setting 
U;-(t, x, Z) Uc (t, kx, kz) 
P(t, x, Z) ®.(t, ky, kz) 


Next, we find control laws for small wave numbers and study uncontrolled wave numbers. 


72.3.1 Controlled Velocity Wave Number Analysis 
Consider k2 + k? < M?. Then x = 1, so control is applied. Using the continuity Equation 72.33 and taking 
divergence of Equations 72.30 through 72.32, a Poisson Equation for the pressure is derived, 

=a? + yy = —4kxiUy(y)V + NVy. (72.40) 


Evaluating Equation 72.31 at y = 0 one finds that 


Vyy (kx 0, kz) 
Re 
ni kxUyo + kz Wyo 
e 


Pylkxs 0, kz) = 


: (72.41) 


where we use Equation 72.33 for expressing Vyy at the bottom in terms of the boundary variables uyg = 
Uy(kx, 0, kz) and Wyo = Wy(k,, 0, kz). Similarly, evaluating Equation 72.31 at y = 1 we obtain 


Vyy(Kxs 1, kz) V, 
yy \x Zz 2%c 
Py(kx, 1, kz) = Re (Vc): — a Re 
kyuy1 + kz W, V, 
ss Higgs (72.42) 
Re Re 


where we use Equation 72.33 for expressing V,, at the top wall in terms of uy = uy(ky,1,k,) and 
Wy1 = Wy(ky, 1, kz) and the controller V.. 
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Equation 72.40 can be solved in terms of integrals of the state and the boundary terms appearing in 
Equations 72.41 and 72.42. 


pa U;(n) sinh (ay — n)) Vikxs 1 k dann f SeheG—w) 


f sagen (a(1 = y)) kytyo + kz Wyo sores: (ay) 


Vo(Rxs to kz) dn 


asinha Re a sinh a 
4ttk,i cosh (a 1 
rile coslt (ay) / U;(n) cosh (a. =n) Vek ns kz) dn 
a sinh a 0 


reas (ay) kyty1 + kzWy1 cg (ay) (Ve4 g2 Ve 
a sinha Re a sinha Re 


ie cosh (a(1 — n)) Vy (kes 1, kz) dy 
0 


(72.43) 


We use the controller V,, which appears inside the pressure solution (Equation 72.43), to make the 
pressure strict-feedback [16] (spatially causal in y), for applying a backstepping boundary controller as 
in [27]. Since the first two lines in Equation 72.43 are already spatially causal, we need to cancel the third, 
fourth, and fifth lines of Equation 72.43. Set 


+ son Rly — Uy) ee ja Wy) 
e 


Cc 


V. 
V = 2 
(Vic)i = Re 


1 
nf cosh (a(1 — n)) Vy(kxs 0, kz) dn 
0 


1 
+4nkdi [ U;(n) cosh (a(1 _ n)) Vik, 1, kz) dy, (72.44) 
0 


which can be written as 


V. ky _ +kz(Wyo — W 
(V.); a 2 Loni (uyo Uy) z( yO yi) NV. 
Re Re 
1 
+] cosh (a(1 _ n)) Viky, 1), kz) (N +4xkiUj(n)) dy. (72.45) 
0 


Then, the pressure is written in terms of a strict-feedback integral of the state V and the boundary 
terms U9, Wyo (proportional to the skin friction at the bottom) as follows: 


pa US (n) sinh (ay — n)) V (Kx 0, k.)dn+N ee 


renee (ay) — cosh (a(1 — y)) 


Re asinha 


Vo (ks to kz) dn 


(kxUyo + kz Wyo) - (72.46) 


Similarly, solving for ¢ in terms of the control ®, and the right-hand side of its Poisson equation 72.34, 


2ni f) sinh (ay) 
b= [sinh (aly — m)) (Keath ty ke) — Be W Che ns b)) dn + 7? 6 (ky) 
2ni sinh } 
ean) / sinh (c(1 —n)) (Ket(kes 0, ke) — ky W (kes 0, ke)) dn, een 
a sinh a 0 


As in the pressure Equation 72.43, an actuator (®, in this case) appears inside the solution for the 
potential. The second line of Equation 72.47 is a nonstrict-feedback integral and needs to be cancelled to 
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apply the backstepping method. For this we use ®, by setting 


2ni f! 
O-(ky; ky) = ee i) sinh (a(1 a n)) (keulks Y)> kz) = ky Wk, Y> kz)) dy. (72.48) 
0 


Then the potential can be expressed as a strict-feedback integral of the states u and W as follows: 


2ni [” 
re sinh (ay _ n)) (keulkx N, kz) — kx Wkx, 1), kz)) dn. (72.49) 
0 


Introducing the expressions (Equations 72.46 and 72.49) in Equations 72.30 and 72.32, we obtain 


cosh (wy) — cosh (a(1 — y)) 


Reasinha 


y 
[F ojton sinh (=n) Voorn dn 


Y sinh = 
—amikn f me) eaten 
0 a 


2 
—aru+u 
ur = re a4 Bu Us (y)V An ky 


(kx uyo +k, Wyo) 


8rk2 
—Nu— x 


An?k,N 
-=—= / sinh (o(y — 0) (KeU (Ke 1, kz) — kW (Ks to ke) dn, (72.50) 
0 


@ h — cosh (a(1 — 
W.= Bw —NW pees (ay) cos (a( y)) 


k kW. 
Re Reasinha (Rettyo + @ v0) 


y 
oa i US(n) sinh (a(y —n)) Vk ke) dn 


Qa 


Y sinh - 
— 2nikew f sr) esa ean 
0 a 
An?k,N [2 . 
+a | sinh (a(y — n)) (kz U (kx, 15 kz) — ke W (kes 0; kz) dy. (72.51) 
0 


We have omitted the equation for V since, from Equation 72.33 and using the fact that V(k,,0, kz) =0, V 
is computed as 


y 
V =—2ni / (kU (kxs 15 kz) + ke W (kes 01 ke) dn. (72.52) 
0 


Now we use the following change of variables and its inverse: 


Y =2ni(kxutk,W), w=2mi(k,u—k,W), (72.53) 
2m 2m 
Defining « = x and the following functions: 
; U; y sinh (ay = 9)) : 
f = 4nik, a5 +f U;(0) do} +Nasinh (aly _ 9) ; (72.55) 
7 a 
h — cosh (a(1 — 
7 cosh (ay) eos (a(1 — y)) (72.56) 
Re sinha 
hy = 2nik,US, (72.57) 


hy = —Nasinh (ay - n)) 5 (72.58) 
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Equations 72.50 and 72.51 expressed in terms of Y and w are 
y 
Y,;=€ (—o?Y ae Gy) —BY —NY +gYyo0 | fk Wo Ns kz) Y (kx 03 kz) dn, (72.59) 
0 


y y 
ws = €(—07@ + yy) =po-No+h [ (keke) an+ f ha(y, nok, ns kz) dn, (72.60) 
0 0 


where we have used the inverse change of variables (Equation 72.54) to express uy and Wyo in terms of 
Yyo = Yy(ky, 0, kz) as follows: 


Yyo = 20 (kyUyo + kz Wyo) » (72.61) 
with boundary conditions 
Y(t, ky, 0, kz) = olt, kx, 0, kz) = 0, (72.62) 
Y (ty ky ly ke) =Yelts kus ke)» (ty kes 1s kz) = We(ty kes ke)s (72.63) 
where 
Yo = 2ni(kyUctkzWe), We = 2m (kzUe — kW) . (72.64) 


Equations 72.59 and 72.60 are a coupled, strict-feedback plant, with integral and reaction terms. A 
variant of the backstepping control design presented in [21] can be used to stabilize the system considered 
here, which consists of a pair of coupled PDEs, by using a pair of backstepping transformations. The 
transformations map, for each k, and k,, the variables (Y, w) into the variables (W, Q), that verify the 
following pair of heat equations (parameterized in k,, kz): 


WY, =€(—w V+ W,,) —BW- NY, (72.65) 
Q = € (—07Q + Qyy) — BA -NQ, (72.66) 
with boundary conditions 
W(ky, 0, kz) = Wlky, 1, kz) = Qk, 0, kz) = Q(ky, 1, kz) = 0. (72.67) 
The transformation is defined as follows: 
v=y-| "K(k 05 ke) ¥ Cs ke) dn, (72.68) 
> z 
Q=0- | Miko mkd¥ Gombe dn— [Palko ynkedolko mk) dn. (72.65) 


The kernel functions K (ky, y, n, kz), Vi(kx, ys 15 kz), and D'2(ky, y, n, kz) are found as the solution of the 
following partial integro-differential equations: 


y. 
éKyy = €Kyy + (B(y) — B(n)) K -f+f f(n,§)K(y,&) dé, (72.70) 
y 
y y 
iy = ing + (BY) —B)) P11 +f Tay, &)h1(&) dé +f fm. 80 1(y, €) dé, (72.71) 
n n 
y 
May = €D2yn + (B(y) — B(n)) P2 — he +f ho(&, n)Pa(y, §) dé. (72.72) 
n 


Equations 72.70 through 72.72 are hyperbolic partial integro-differential equation in the region T = 
{(y.n):0<y <1,0<1n < y}. Their boundary conditions are 


y ‘ou 
Oe a K(,0) = So Seana gy) ne 
‘y 
Nyy) =90, Pily,0) = Jo Ti.) dn (72.74) 


€ 
T2(y, y) = 0, T2(y, 0) = 0. (72.75) 
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Remark 72.2 


Equations 72.70 through 72.75 are well-posed and can be solved symbolically, by means of a successive 
approximation series, or numerically [21]. Note that Equations 72.70 and 72.72 are autonomous. Hence, 
one must solve first for K(kx, y, n, kz) and '2(kx, y, 1, kz). Then the solution for I’2 is plugged in Equation 
72.71 which then can be solved for Ty (ky, y, 1, kz). 

Control laws Y, and W, are found evaluating Equations 72.68 and 72.69 at y = 1 and using Equations 
72.63 and 72.67 which yield 


1 
Yee = if K(kes ty ke)¥ (kes 1p ke) dn, (72.76) 
0 


1 1 
W(t, ky, kz) =[ V1 (ky, 1, YN, kz) Y (kx, mke)an+ f T2(ky, 1, Ns kz)wlky, Y> kz) dyn. (72.77) 
0 0 


Using Equations 72.53 and 72.54 to write Equations 72.76 and 72.77 in(u, W), we obtain 


1 a 
tee / KY (key 1st ke)u(Kes Ws ke) dy + / KOW (es 1. 1h ke) W (ken 1» ke) a, (72.78) 
0 0 
1 1 
W.= i K™#(ky, 1, 1 ke)(kas Nhe) dn + if K’ Wek Lavk)Wkemke)dn, (72.79) 
0 0 
where 
oe K(k, ys 1 kz) 
K 2 Ty (kys ys 1 kz) 
xwu | =A 0 ; (72.80) 
Kww Polke. 941, ke) 


and where the matrix A is defined as 


a aan k2 
4m? [kyk, 2 = =kk 
eo [kk —k ke kk, 

ke Ses Skike. 


(72.81) 


Stability in the controlled wave number range follows from stability of Equations 72.65 and 72.66, and 
the invertibility of the transformations (Equations 72.68 and 72.69). We obtain the following result. 


Proposition 72.1: 


For k2 +k2 < M’, the equilibrium u= V = W = 0 of systems (Equations 72.30 and 72.37) with control 
laws (Equations 72.45, 72.48, 72.78 and 72.79) is exponentially stable in the L” norm, that is, 


d; 1 
[ (uP +1VP-+1WP) Osho rhe) dy = cae (lui? +1V)? +1WI*) (0, kes yke) dy, (72.82) 
0 0 


where C, > 0. 
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Proof. From Equations 72.65 and 72.66 we obtain, using a standard Lyapunov argument, 


1 1 
[WP +19?) (ko keddy se f (WI? +1217) (0, kes ys kz) dy, (72.83) 
0 0 


and then from the transformation (Equations 72.68 and 72.69) and its inverse (which is guaranteed to 
exist [21]), we obtain 


1 1 
i (IV 2+ lol?) (ts ken yoke) dy < Coe! i: (IY2-+ 101) Okeke) dy, (72.84) 
0 0 


where Cp > 0 is a constant depending on the kernels K, I}, and Iz and their inverses. Then writing 
(u, W) in terms of (Y, w) and bounding the norm of V by the norm of Y (using Y = —V, and Poincare’s 
inequality), the result follows. 


72.3.2 Uncontrolled Velocity Wave Number Analysis 


When k? + k? > M, the plant verifies the following equations: 


—o7u+ Uyy é ; . 
ur; = = Bu Uy (y)V — 2akxip + 20k,iNo — Nu, (72.85) 
—0?V + Vyy 
Vi= Re BV — py, (72.86) 
—o?W+ Wy 
t= R BW — 2k, ip — 20k,iNo — NW, (72.87) 
e 
the Poisson equation for the potential 
—a7h + dyy = 2m (kzu— k, W) (72.88) 
the continuity equation 
2nikyu+ Vy + 2nk,W =0, (72.89) 
and Dirichlet boundary conditions 
u(t, kx, 0, ky) = V(t, ky, 0, ky) = W(t, ky, 0, ky) = 0, (72.90) 
u(t, ky, 1, ky) = V(t, kx, 1, ky) = W(t, kx, 1, ky) =0, (72.91) 
O(t, kx, 0, ky) = O(t, kx, 1, ky) = 0. (72.92) 


Using the transformation (Equation 72.53) to write the system in (Y, w) coordinates, one gets the following 
equations for Y and w: 


¥; =€(—07Y + Yyy) — BY — 20k,iUy V + 07p— NY, (72.93) 
w = €(—a7w + wy) — Bo — 20k,iUyV — a Nb — No. (72.94) 
The Poisson equation for the potential is, in terms of w, 
=a + by =O. (72.95) 
Consider the Lyapunov function 


1 1,12 2 2 
V Ww 
A =| ae dy, (72.96) 
0 


where we write /, jc : f (kes Ys kz) dy. The function A is the L? norm (kinematic energy) of the velocity 
field. 
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Substituting Y and w from Equation 72.54 into Equation 72.96, we obtain 


1 FP RIYP+RloPr +hk(Yo+tYo)  RIY?P+Rl ol? —kyk,(Yo+ YO) 
A= F Ant 2d + Qt dy 


Lyy)2 2 212 
Y 4 
=) EAL cada (72.97) 
0 202 
Define then a new Lyapunov function, 
1 Y 2 2 2 V 2 
Aswan [ Ele ole dy. (72.98) 
6 2 
The time derivative of A, can be estimated as follows: 
1 1 
Ay = —2e07 Ay — ef (IYyI? + lol? +071V,/7) -vf (IY? +l’) 
0 0 
1 7 1 iz 
=| iUSQ)V Oks + ket) + miU,(y) V (2kxY + kzw) 
0 0 
16 o 1 pY+PY—P,V—P,V 
= en | DON DD 28 i sii Als (72.99) 
0 2 0 2 
For bounding Equation 72.99, we use the following two lemmas. 
Lemma 72.1: 
2 [) oa+o eee 
~a / Puy ee < | ||. (72.100) 
0 2 0 
Lemma 72.2: 
|US()| <4+H. (72.101) 
Integrating by parts and applying Lemma 72.1, 
. 1 1 - 
Ay < 2607 Ay -ef (I¥yl? + lwyl? + 071 Vy”) +{ MiUy(y)V (kx Y + kz) 
1 _ 1 
-| miUSYVU¥ +ko)—N f ly/. (72.102) 
0 0 


Using Lemma 72.2 to bound U; in Equation 72.102, 


1 1 
Ay s—2e(14+02) Ar —w f IyPady + 2m (444) | (IV (kel LY] + [kz ||) dy 
0 0 


< (4+H—-2(1+0")) Ai (72.103) 
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where we have applied Young’s and Poincare’s inequalities. Hence, if a” > ‘tH ; 
A, <—2€A). (72.104) 
Dividing Equation 72.104 by a? and using Equation 72.98, we obtain 
A <—2«€A, (72.105) 


and stability in the uncontrolled wave number range follows when k? + k? > M? for M (conservatively) 
chosen as 
ifs A Gs: 4)Re 
2b 2 


We summarize the result in the following proposition. 


(72.106) 


Proposition 72.2: 


For k2 +k? > M?, where M > a ERs, the equilibrium u = V = W = 0 of the uncontrolled systems 
(Equations 72.85 through 72.92) is exponentially stable in the L” sense, that is, 


1 1 
[ (uh +1VP +1WP) ha ykerdy se | (Jul? +1VI7 +1WI?) (O,kesy,ke) dy. (72.107) 
0 0 


72.3.3. Closed-Loop Stability Properties 


Substituting Equations 72.45, 72.48, and Equations 72.78, 72.79 into Equation 72.39, and using the Fourier 
convolution theorem, we obtain the control laws in physical space, which can be expressed compactly as 


U; lee) 1 [e-e) 
wJ=f ff Be m2-0( ween) a dn dt, (72.108) 
oo, eda WE no) 
where 
CO CO P 
Ben) =f / Likes Ns ke) X (Kees ke ertlhsS +k) dk, dkx, (72.109) 
—oo J—00 
and 
Kee yn ke) KW (ky, 1, 1, kz) 
Pel K Mk. Lk) KYM hes ink), (72.110) 
2nikz sinh(a(1—n)) 2nik, sinh(a(1—n)) 
Qa Qa 


where the kernels appearing in Equation 72.110 were defined in Equation 72.80. Control law V; is a 
dynamic feedback law computed as the solution of the following forced parabolic equation: 
(Ve)xx + (Ve)zz 


(Ve)t = R NV. + g(t, x,Z); (72.111) 
e 


where g(t, x, Z) is defined as 
oo oo 1 
c= / | f gv(x—& 0,2 —6)V(E, 0,6) dy + gw(x —&,z—6) (Wy(E, 0,6) 


—Wy(E, 1,8)) + gu(x — &, 2 —b) (uy(E, 0, t)—uy(E, 19) dé dt, (72.112) 
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and 

le) lee) ky ; 

w= | l Ami X (kes kee EHD) dky dk, (72.113) 
—coJd—co Re 
[oe Co : 

eve i / cosh (a(1 — )) (N +4xkiU;(n)) XKys kelermlhe +k) dk, dk, (72.114) 
oe | 

gw= / / Ami y (kes kee? +89) dk, dk. (72.115) 
—coJd—co Re 


Considering all wave numbers and using Propositions 72.1 and 72.2, the following result holds regard- 
ing the convergence of the closed-loop system. 


Theorem 72.1: 


Consider the systems (Equations 72.18 through 72.27) with control laws (Equations 72.108 through 72.115). 
Then the equilibrium profile u= V = W = 0 is asymptotically stable in the L? norm, that is, 


lo) 1 lo) 
/ i / (uw +V74 w’) (t,x, y,z) dx dy dz 
—wJd0 —0o 
lo) 1 lo) 
< aa | i / (uw 474 w’) (0, x,y,z) dx dy dz. (72.116) 
—oo J0 —0o 
where Cp = max{Cj, 1} > 0. 


Remark 72.3 


In case that N = 0, meaning that either there is no imposed magnetic field or the fluid is nonconducting, 
Equations 72.2 through 72.4 are the Navier-Stokes equations and our controller solves the stabilization 
problem for a 3D channel flow. Some physical insight can be gained analyzing this case. In the context of 
hydrodynamic stability theory, the linearized system written in (Y, w) variables verify equations analogous 
to the classical Orr-Sommerfeld-Squire equations. These are Equations 72.59 and 72.60 for controlled 
wave numbers and Equations 72.93 and 72.94 for uncontrolled wave numbers. Note that we use the 
backstepping transformations (Equations 72.68 and 72.69) not only to stabilize (using gain K) but also to 
decouple the system (using gains I), 7) in the small wave number range, where nonnormality effects are 
more severe. Even if the linearized system is stable, nonnormality produces large transient growths [19], 
which enhanced by nonlinear effects may allow the velocity field to wander far away from the origin. This 
warrants the use of extra gains to map the system into two uncoupled heat equations 72.65 and 72.66. 


72.4 Observer Design 


In this section we design an observer for the MHD channel flow introduced in Section 72.2. Our observer 
generates estimates of the velocity, pressure, electric potential, and current fields in the whole domain, 
derived only from wall measurements. Obtaining such an estimate can be of interest in itself, depending 
on the application. For example, the absence of effective state estimators modeling turbulent fluid flows 
is considered one of the key obstacles to reliable, model-based weather forecasting. In other engineering 
applications in which active control is needed, such as drag reduction or mixing enhancement for cooling 
systems, designs usually assume unrealistic full state knowledge, therefore, a state estimator is necessary 
for effective implementation. 
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72.4.1 Observer Structure 


For simplicity, we first design an estimator for the linearized system. In Section 72.4.6, using the linear 
gains, we present a nonlinear observer. 

We employ the fluctuation variables around the equilibrium of the Hartmann flow, u and p, which 
were defined in Equations 72.16 and 72.17; the linearized equations written in fluctuation variables are 
given in Equations 72.18 through 72.27. 

The observer consists of a copy of the linearized equations, to which we add output injection of the 
pressure p, the potential flux $y (proportional to current), and both the streamwise and spanwise velocity 
gradients, uy and Wy, (proportional to friction) at the bottom wall. 

Denoting the observer (estimated) variables by a hat, the equations for the estimated velocity field are 


Au 


iy = S— — U*(y)itx — UV Px + No, —Nia— QY, (72.117) 

e 

a2 VE #6 on sa 

Vi= ao — U%(y) Vz — py — Q”, (72.118) 
e 

~ AW tier of : P 

Wi = = — U'Q)We pz -Nbx -NW-—Q". (72.119) 
e 


The additional Q terms in the observer equation are related to output injection and are defined as follows: 
p(§) 0,6) — p(&, 0,6) 


QU Pp 
eh fps 7 7 Uy(§, 0,6) — ay (, 0,6) 
ow Sif [. L(x — §, y,z—¢) W,(E,0,t) — Wy(8, 0,0) dé dt, (72.120) 


y(E, 0,8) = by, 0,0) 
where L is an output injection kernel matrix, defined as 
LUP LUY LUwW Lue 
L= LY? LYY LYW LY® 
LWP LW LWw LWo 


‘ (72.121) 


whose entries will be designed to ensure observer convergence. The estimated potential is computed from 


Ad = iz — We, (72.122) 
and the observer verifies the continuity equation, 
ity + Vy + Wz =0, (72.123) 
and the same boundary conditions as the plant, 
iu(t, x, 0,z) = W(t, x, 0,z) = V(t, x, 0,z) = b(t, x, 0,z) = 0, (72.124) 
in(t, x, 1,z) = Ue, W(t,x,1,z) = We, (72.125) 
V(t,x,1,z) = Ves b(t, x, 1,z) = ®e. (72.126) 


Remark 72.4 


Note that the observer Equations 72.117 through 72.126 can be regarded as forced MHD equations, with 
the output injection acting as a body force. This means that any standard DNS solver for the forced MHD 
equations can be used to implement the observer without the need of major modifications. 

As inputs to the observer, appearing in Equation 72.120, one needs measurements of pressure, skin 
friction, and current on the lower wall. For obtaining these measurements, pressure, skin friction, and 
current sensors have to be embedded into one of the walls. Pressure and skin friction sensors are common 
in flow control, while for current measurement one could use an array of discrete current sensors, as 
depicted in Figure 72.3. 
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Current density sensor array 


FIGURE 72.3 An array of current sensors in the lower wall. 


72.4.2 Observer Gain Design and Convergence Analysis 


Substracting the observer equations from the linearized plant equations we obtain the error equations, 
with states U =u—u=U-—U,V=V—V,W=W-—W,P=p-—f,andb= 6-9, 


AU - ~ = 7 ~ 

U, = S- — UUs — US — Pe + Nb: — NO + Q", (72.127) 
e 

~ AV a 

i= Fy UO Py +, (72.128) 
e 

~ AW 28s m * 

Wises <= U*(y) Wy — P, — No, -NW+Q”. (72.129) 
e 


The observer error verifies the continuity equation, 
Uy + Vy + Wz =0, (72.130) 


while the potential error is governed by 
Ao = U;, — Wy. (72.131) 


The boundary conditions for the error states are 


U(t, x, 0,z) = V(t, x, 0,z) = W(t, x,0,z) =0, (72.132) 
U(t, x, 1,z) = V(t, x, 1,z) = W(t,x, 1,z) =0, (72.133) 
b(t, x, 0, z) = (t,x, 1,z) = 0. (72.134) 


To guarantee observer convergence, our design task is to design the output injection gains L defined 
in Equation 72.120 that appear in QY, QY, and Q™, so that the origin of the error system is exponentially 
stable. 

Using the Fourier transform in x and z as defined in Equations 72.28 and 72.29, we obtain the observer 
error equations in Fourier space, which are 


‘ti 
% —aU+U, z = Le FS _ 

L= =  — BU — USV — 2mkyiP + 2nk,iNO — NU 

+ LP Py + LY Uy + LOW Wyo + 150, (72.135) 
pein & 

as —a’V+ V, ee 

Ve, BV — Py + LP Py + LYE Uy + LY Wyo +L" yo, (72.136) 
a —o? W + Wyy ~ vas ae S 
W, = Fr BW — 2nk,iP — 2nk,iNb —-NW 

e 


+ LP Py + LY Uy FLY Wyo +L yo, (72.137) 
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where we have used the definition (Equation 72.120) of the output injection terms as convolutions, 
which become products in Fourier space. We have written for short Py = P(kx, 0, kz), Uyo = Uy(ky, 0, kz), 
Wyo _ Wy(kx, 0, kz), and byo = by(kx; 0, kz). 

The continuity equation in Fourier space is expressed as 


2nik,U + Vy + 20k,W = 0, (72.138) 
and the equation for the potential is 
~a7 + byy = 2mi (ke — kW). (72.139) 


Note that Equations 72.138 through 72.143 is uncoupled for each wave number. Therefore, as in 
Section 72.3, we define the range k2 +k? < M? as the observed wave number range, and the range 
k2 + k2 > M? as the unobserved wave number range, and study them separately. If stability for all wave 
numbers is established, stability in physical space follows as in Section 72.3.3. A bound for the number M 
was given by Equation 72.106 to ensure stability for the unobserved wave number range. 

We emphasize that we do not use output injection for unobserved wave numbers by writing 


L(kx, Ys kz)= Kk Vs kz)Rlkxs ys ky), (72.140) 


where x was defined in Equation 72.38. Then L can be written in physical space, using the definition of 
the Fourier transform and the convolution theorem as 


CO CO 
L(x, y,z) = / / X (kes Yo ke)R Kes ys kze2tlker +2) dk, dk. (72.141) 
—0o J—0O 


The matrix R is defined as 
RUP RUU RUW RYO 


Rae) Re Re ORV RNe hs (72.142) 
RWP RWU RWW pWo 
and using R we can write the observer error equations as 
v7 +U, ~ ~ ~ 7 5 
(= a ” — BU — UV — InkyiP + 2nk,iNb — NU 
e 
+X (kes kz) {ROP Po + ROY Uy + REY Wyo + RY yo} 5 (72.143) 
~ —wWV+V. ~ 
ne BV — Py +X (Kkeske) {RYPPo + RV Uyo +R WyoR yo} , (72.144) 
% —a? W + Wyy is 8 he, * 
WwW, = re BW — 21k,iP — 20k,iNbd —NW 
+X (kx kz) {RY PPo + RY Uy +R Wyo +R? yo} . (72.145) 


72.4.3 Observed Wave Number Analysis 


Consider k2 + k? < M?. Then x = 1,80 output injection is present. Using the continuity Equations 72.138 
and 72.143 through 72.145, the following Poisson equation for the pressure is derived: 


—07 P+ Py = Y—AnksiU;(y)V + NVy, (72.146) 
where Y contains all the terms due to output injection, 
T=Py (27% RY +RVP 4 2nk-RW?) iy (271k ROU + RV 4 2nk-RW”) 


+ Wyo (2mk, ROW 4 RYW 2nk:RW ) + by0 (2mik, RU +RVO4 2nk,RW*) _ (72.147) 
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We want to make Equation 72.146 independent of the output injection gains, for which we need Y = 0. 
Hence, we set 


RV? (ky, y, ke) = RY? (ky, 0, ke) — 200i [ i (ke RU? + kR””) (ky, 03 ke) dn, (72.148) 
RY (ky yk) = RY (ky, 0, kz) — 200 [ x (keROY + kRWY) (kes ke) dn, (72.149) 
RY (ky, yy ke) = RY (ky, 0, ke) — 200i i, Z (keROW + k,RW) (ks 0, kz) dn, (72.150) 
RV °(k,, y, kz) = RY? (ky, 0, kz) — 200i ix (keRU? + keR®) (kes kz) dn, (72.151) 


which means that, in physical space, V - L = 0. Hence, as Equations 72.146 is derived by taking divergence 
of Equations 72.143 through 72.145, the output injection terms cancel away. 
Expression 72.146 can be solved in terms of the values of the pressure at the bottom wall. 


4nk,i [’ __. . 2 
A i U;(n) sinh (ay _ n)) Viky, N, kz) dy + cosh (ay) Po 
0 


sinh (ay) S Y sinh (aly = n)) 
Qa i a 


P=- 


+ Py(ky, 0, kz) +N Vy (kes 3 kz) dn. (72.152) 


Evaluating Equation 72.144 at y = 0, one finds that 


: ky Uyo (ky, 0, kz) + k, Wyo 
1 
Re 


Py (kx, 0, kz) = Yo — 20 : (72.153) 


where we have used Equation 72.138 for writing Vy at the bottom in terms of measurements. In Equation 
72.153, 


Yo = PoRY” (kx 0, ke) + UyoR’” (kes 0, ke) + WyoRY™ (kxs 0, kz) + by0R” (kx 0, kz), (72.154) 
and as before we force the pressure to be independent of any gains. Hence, we set 
RY? (ky, 0, kz) = RY (ky, 0, ke) = RY (ky, 0, kz) = RY? (ky, 0, kz) = 0. (72.155) 


Then, the pressure can be expressed independently of the output injection gains in terms of a strict- 
feedback integral of the state V and measurements, 


is 4tkyi [? __. : is 
PpP=-— 5 [ Uy (n) sinh (cx(y — n)) V (ky, 1, kz) dy + cosh (ay) Po 
inh Y sinh - 4 
3) (kx Uyo + kz Wyo) +n f SEE GUS ee haa (72.156) 
ea 0 Qa 


Similarly, solving for > in terms of the measurement ¢yo and the right-hand side of Equation 72.139, 


b= = i sinh (a(y — n)) (kz U (kes 05 ke) — ke W (kes 03 kz) dy + Se) (72.157) 
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Introducing the expressions (Equations 72.156 and 72.157) in Equations 72.143 and 72.145, we obtain 


~ —I+0, : ee aes 4n?k2 
= * — BU — Uy()V — NU + Uy (av - — sinh («)) 
Quine 


Re 
sinh («)) 
21k 8k? 


; ji : 
+ by0 (x +N== sinh («)) ca x U;(n) sinh (u(y — n)) Vikxs 15 kz) dn 
0 


2 
+ Po (RY? — 2nk,i cosh (ay)) + Wyo (Rw _ ans 
a 


An*k,N [? s s 
—— / sinh (ce(y — 1) (kel (kes ts ke) — ke W (kes ty ke) dy 
0 


Y sinh (a(y — n)) ~ 
— 2mikyN i stan) Fe spe (72.158) 
0 a 
“s —-WV@W+W PS e An?k,k 
yY WU _ xKz 
W, = a BW -—NW +4 Ujo (k Re sinh )) 


+ Po (RW? — 2xkzi cosh ww _ 4mtke 
0 ttkzi cos (ay)) + Wyo R She sinh (ay) 


80k,k, 
a 


2tkyi ? ; is 
+ byo (a -n== sinh ()) _ / U;,(n) sinh (aly — n)) Vik, , kz) dy 
0 
An2kyN [2 . is . 
4 EE [sini (ety =m) (Re Os. ke) = bck oe) 
0 
yan a= ai) 
— 2nik,N / sna) Pee ean (72.159) 
0 Qa 


Note that we have omitted the equation for V since, from Equation 72.138 and using the fact that 
V (kx; 0, kz) = 0, V is computed from U and W: 


g Vee a 3 
v= -2ni f (kU (kxs 13 kz) + kz W(kxs 1, kz)) dn. (72.160) 
0 


We now set the output injection terms to directly cancel the boundary terms coming from Equations 
72.156 and 72.157, while still leaving some additional gains for stabilization. Thus, we define 


RUP — nk, i cosh (ay) ,RWP 2nk,i cosh (ay) : (72.161) 
An? k? 4n?k,k 

RY — = * sinh (wy) + Ti(ky, kz), RNY = a sinh (ay) + Ta(kx, ys kz), (72.162) 

aRe e 

An*k,k 4n?k2 

RUW ue Esai e te pko ke" = — sinh (ay) + Tla(kes ys kz)s (72.163) 

2nkzi Qk, 
RYO — yn sinh (ay) , RW¢— nN sinh (ay), (72.164) 
a a 


where the gains IT), Iz, I3, and Iq are to be defined later. From Equations 72.148 through 72.151, 
72.155 and 72.161 through 72.164, we obtain an explicit expression for the remaining entries of R, 


RY? =asinh (ay), RY? =0, (72.165) 
1—cosh y 
RY = 2ni(ky + Qe) = anif (ke Ti (kes 1 ke) + keTla(kes 1, ke) dn, (72.166) 
e 0 
1—cosh y 
RYW = 2ni(k, + jw) = 2nif (ke E13 (kys 1 ke) + keTLa(kes ny ke)) dn. (72.167) 
e 0 
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Introducing Equations 72.161 through 72.167 in Equations 72.158 and 72.159, we obtain 


- —wt+Uyy 
Pes Re 
An?k,N es 28 
“3 es i, sinh (aly _ n)) (k,U(kx, Ns kz) — ky Wk, 1) kz) dy 
0 


81k? 


BU Us(y)V — NU + My Uyo + 113 Wyo 


y ss 
[Pe feay sinh (at — ) Hobson ke) a 
0 


adh Gas 
— 2nik,N i SO ait Nai, (72.168) 
0 Q 


Pee 
5 —atW + Wyy 


W; = 


BW —NW + I1,Uj0 + M4 Wyo 

= Sri ik Uy(n) sinh (ay — n)) Vikxs ns ke) dn 
+ —_ [ sinh (a(y — n)) (kz U(kxs 13 kz) — ke Wks 0 kz) dn 

—2nik,N if SEEN cae) dn. 72162) 


Now, we introduce the following change of variables and its inverses: 


Y =2ni(k,U+k,W), w=2ni(k,U—k,.W), (72.170) 
2ni 2ni 
U= > (kxY+ko), W= > (kzY — kw). (72.171) 


Then using the definitions in Equations 72.55 through 72.58, Equations 72.168 and 72.169 expressed in 
terms of Y and w are 


An? 
Y; =€(-a°¥ + Y,,) —BY -NY— ae (KqT1) + keke M2 + keke 13 +k2 14) Yyo 
4n? 2 2 y 
on (keke Ty + k2 M2 — k113 — kk, 114) wyo +f Ff (kes ¥os kz) ¥ (kes 1 kz) an, (72.172) 
0 


An? 
@, = €(—aw + wyy) — Bo — No — <> (kekeTh — kilo + k2113 — kykzT4) Yyo 


4n? y 
2 (KET ~ kha ke Ta + K3Ts) 30 +'n(9) [YC mo ke) dn 
0 
oA 
+f holy, nwlkx, 0, kz) dn, (72.173) 
0 


where we have used the inverse change of variables (Equation 72.171) to express Uyo and Wyo in terms of 
Yyo = Y (ky, 0, kz) and wyo = w(kx, 0, kz). We define now the output injection gains I11, M2, M3, and I, 
in the following way: 


Ty I(kx, y, 0, kz) 
TI, Ps | 0 
=A 72.174 
113 91 (kx ys 0; kz) 
Tl4 Oo(kx, y, 0, kz) 
where the matrix A was defined in Equation 72.81. Note that since det(A) = —1 its inverse appearing 


in Equation 72.174 is well defined. The functions I(k,,y,n,kz), 91(ky,y, kz), and 92(ky, y, 1, kz) 
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in Equation 72.174 are to be found. Using Equation 72.174, Equations 72.172 and 72.173 become 


BA 

Y, =€( aY 4 Yn) BY —NY4 Kk 04K.) ¥0+ f S kes Ys Ms kz) Y (kx 1 kz) dn, (72.175) 
0 

w, = € (—a?w + @yy) — Ba —Nwt 91 (ky, y, 0, kz) Yyo + 92(kxs y, 0, kz)@yo 


BA y 
+h / ¥ (kes ke) dn + i holy, nealkes ke) dn. (72.176) 
0 0 


Equations 72.175 and 72.176 are a coupled, strict-feedback plant, with integral and reaction terms. A vari- 
ant of the design presented in [22] for anticollocated systems can be used to design the gains I(k,, y, 0, kz); 
01 (kx; y, 0, kz), and 02(k;, y, 0, kz) using a pair of backstepping transformations. The transformation maps, 
for each kx and kz, the variables (Y, w) into the variables (W, Q), that verify the following pair of heat 
equations (parameterized in k,, kz): 


W; =€(-o° W + Wy) —BW- NW, (72.177) 
Qy = € (—a’Q+ Qyy) — BQ —NQ, (72.178) 

with boundary conditions 
Wkx, 0, kz) = Wik, 1, kz) = Qk, 0, kz) = Qky, 1, kz) =0. (72.179) 


The transformation is defined as follows: 
y 
Y= vf I(kys V5 Ns kz) Wlkxs 1, kz) dn, (72.180) 
0 
y yy 
O= a- f 91 (kx, Ns kz) Vk, 0, kz) dy =} 92(kxs VN» kz) Q2(kx, N, kz) dn. (72.181) 
0 0 


We find the kernel functions [(k,, y, 1, kz), 91(kx, ys 0, kz), and 92(kx, y, 1, kz) by solving the following 
partial integrodifferential equations: 


y 
lay = hy — (8G) —B(n)) FF + i. FO, 8)ICG,n) a, Gz 
a) 


y y 
€O1nn = O1yy — (B(y) — B(n)) O1(y, 0) — Ay + hy / I(§, n) d& +f ha(y,§)01(§,n)d§, (72.183) 
n n 


yy 
Bonn = €O2yy — (By) — B(n)) 02 — ha + / ho(ys)2(E)n) dé. (72.184) 
n 


Equations 72.182 through 72.184 are hyperbolic partial integro-differential equation in the region T = 
{(y.n):0<y <1,0<1n < y}. Their boundary conditions are 


[kx Vs Ys kz) = O1 (kx, VV k,) = O2(kxs Vs Vs kz) =0, (72.185) 
Ik, 1, 0, kz) = 91 (kx, 1, 0, kz) = 92(ky, 1, 1), kz) = 0. (72.186) 


Remark 72.5 


Equations 72.182 through 72.186 are well-posed and can be solved symbolically, by means of a successive 
approximation series, or numerically [21]. Note that both Equation 72.182 and Equation 72.184 are 
autonomous. Hence, one must solve first for I(k,, y, 7, kz) and 92(k,, y, 1, kz). Then the solution for / is 
plugged in Equation 72.183 which then can be solved for 6 (kx, yn, kz). The observer gains are then 
found just by setting 7 = 0 in the kernels I(k,, y, n, kz), 92(kxs ys Ns kz), and 91 (ky, y, 15 kz). 

Stability in the observed wave number range follows from stability of Equations 72.177 and 72.178 and 
the invertibility of the transformations (Equations 72.180 and 72.181), as in Proposition 72.1. 


72-29, Control System Advanced Methods 


72.4.4 Unobserved Wave Number Analysis 


When k? + k2 > M, there is no output injection, as x = 0, and the linearized observer error verifies the 
following equations: 


as —wU+ yy © = oe nee ~ 

Up, = > BU U;(y)V — 21kxiP + 20kziNb — NU, (72.187) 
e 

ix —2V+ Vv wise’ ab 

YY 

v= = BV — By, (72.188) 

~ —wWW+Wy e ee - 

Wi= 3 BW — 21k iP — 21k,iNb —NW, (72.189) 
e 


the Poisson equation for the potential Equation 72.139 and the continuity Equation 72.138. 

Note that Equations 72.187 through 72.189 are the same as Equations 72.85 through 72.87. Hence, the 
analysis of Section 72.3.2 can be applied, obtaining a result similar to Proposition 72.2. Hence, stability in 
the unobserved wave number range follows when k? + k2 > M? for M as in Equation 72.106. 


72.4.5 Observer Convergence Properties 


Considering all wave numbers, the following holds regarding the convergence of the observer. 


Theorem 72.2: 


Consider the systems (Equations 72.18 through 72.27), and the systems (Equations 72.117 through 72.126), 
and suppose that both have classical solutions. Then, the L? norms of U, V, and W converge to zero, that is, 


ee) 1 ee) 
lim / [ / (v? +074 Ww’) (t,x, y,z) dx dy dz =0. (72.190) 
—oo J0 —0o 


t>oo 


Remark 72.6 


The convergence result stated in Theorem 72.2 guarantees asymptotic convergence of the estimated states 
to the actual values of the linearized plant. For this to be true for the nonlinear plant we need additional 
conditions. Namely, the estimates have to be initialized close enough to the real initial values and the 
MHD system has to stay in a neighborhood of the equilibrium at all times. 


Remark 72.7 


In case that N = 0, meaning that either there is no imposed magnetic field or the fluid is nonconducting, 
Equations 72.18 through 72.20 are the linearized Navier-Stokes equations and the observer reduces to 
a velocity/pressure estimator for a 3D channel flow. This is a result that can be seen as dual to the 3D 
channel flow control problem. See Remark 72.2 for some physical insight for this case. 


Remark 72.8 


We obtain an output feedback law that stabilizes the plants (Equations 72.18 through 72.27) using only 
wall measurements. Such a control law uses the estimates (i, V, W) from the observers (Equations 72.117 
through 72.126) to replace the real states (u, V, W) in the control laws (Equations 72.108 through 72.115). 
Then, using Theorems 72.1 and 72.2 and standard arguments for linear output feedback controllers, a 
similar result holds guaranteeing the L” stability of the closed-loop output feedback system. 
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72.4.6 A Nonlinear Estimator with Boundary Sensing 


We present a nonlinear observer to obtain convergence in open-loop estimation problems (the linear 
estimator achieves convergence in feedback problems, where both the state and the estimate are driven 
to zero, however, the linear observer, which neglects the nonlinear terms in the model, cannot capture 
open-loop dynamics, and hence cannot achieve convergence to the solutions of an uncontrolled system). 
The nonlinear observer has the same structure and gains as the linear observer, but the nonlinear terms 
are added. In our design, we follow an approach similar to an Extended Kalman Filter, in which gains are 
deduced for the linearized plant and then used for a nonlinear observer. 
The nonlinear observer equations are the following: 


~ AU 


‘=z UU, — VU, — WU, — P, + No, —NU- QY, (72.191) 
PS) ee ee ee ee 
r= UM VV, — WV, By —, (72.192) 
Ro WW ces ee i. . fi sae 
Wi= = — OW, — VW, — WW, — B, —Nb,-NW—Q". (72.193) 
e 


The estimated potential is computed from 
Ad = U, — Wy; (72.194) 
and the observer verifies the continuity equation, 
0, +V,+W,=0, (72.195) 


and the same boundary conditions as in Equations 72.124 through 72.126. 

In Equations 72.191 through 72.193, the Q terms are the same as for the linear observer. Hence, the 
observer is designed for the linearized plant and then the linear gains are used for the nonlinear observer. 
Such a nonlinear observer will produce closer estimates of the states in a larger range of initial conditions. 

Using the fluctuation variable and the observer error variables, we can write the nonlinear observer 
velocity field error equations as follows: 


AU 


= a: Uy), +N“ (0, V, Wu, V, W) — Us) V — Py + NO, —NO+Q, (72.196) 
eé 

Be IV i 1S Vie od a: 

Wwe5,7U (y)Vx +N" (U,V, W,u, V, W) — Py + Q", (72.197) 
e 

a AW e a Witr vu tn B 7 v1 Ww 

Wr = — UO)We t NNO, V, Wu, Vs W) — Pz — Nox — NW +Q™, (72.198) 


NY = 00, —u, — Ou, + VU, — VU, — Vu, + WU, — WO, - Wu, (72.199) 
NV = UV, —uV,— UV, + VVy —VVy— VV) + WV, - WV, - WVz, (72.200) 
NW =UW, -uW, -—UW,+ VW, — VW, - VW, + WW, - WW, - WW, (72.201) 


Assuming, for the purposes of observer design and analysis, that the observer state (U, V, W) is 
close to the actual state (U,V, W) (ie., the error state is close to zero), and that the fluctuation 
(u, V, W) around the equilibrium state is small, then Nu (U, V, Wu, V, W), NV(U, V, W, 0, V, W); 
and Nw(U, V, W, u, V, W) are small and dominated by the linear terms in the equations, so they can be 
neglected. The linearized error equations are then 

2 NU 


U1 = S> — UU — US — Pe + Nb: —NO + Q", (72.202) 
e 
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Vi= Zp UO Py + (72.203) 
e 

~ AW Bien Os ~ 7 

Wi = S — Uy) We — Pe - Noy — NW + on, (72.204) 
e 


which are the same as Equations 72.127 through 72.129. Thus, as expected, the error equations for the 
observer designed for the linearized plant, and the linearized error equations for the nonlinear observer 
are the same; this is the main reason why the same gains derived in Section 72.4.2 are used. 


Remark 72.9 


Following [10], we may consider the mean turbulent profile instead of considering the exact laminar 
equilibrium profile. This amounts to changing U® in definition (Equation 72.12). Since U* appears in the 
coefficients of the output injection gain PDEs (Equations 72.182 through 72.184), the observer gains will 
change (quantitatively). However, Theorem 72.1 still holds and guarantees convergence of estimates, but 
for these estimates to be good enough it is required that the state has to stay close enough to the mean 
turbulent profile at all times. 


72.5 For Further Information 


We survey some representative results in flow control, a recent but rapidly growing field. 

By far the most studied problem in flow control is channel flow stabilization for large Reynolds 
numbers. There are many complex issues underlying this problem [13,14,19], making it extremely chal- 
lenging. Optimal control has so far been the most successful technique for addressing channel flow 
stabilization [13], in a (streamwise- and spanwise-) periodic setting, by using a discretized version of the 
equations and employing high-dimensional algebraic Riccati equations for computation of gains. Using 
a Lyapunov/passivity approach, another control design [1,3] was developed for stabilization of the (peri- 
odic) channel flow. Other works make use of nonlinear model reduction techniques to solve the problem, 
though they employ in-domain actuation [5]. Boundary controllers using spectral decomposition and 
pole-placement methods have been developed [25]. Other techniques include separation control [2] and 
turbulence suppression by using transverse wall oscillations [15]. 

Observer design has been so far a largely neglected area in flow control. For channel flows it is known 
that pressure and skin friction at one of the walls completely determine the flow inside the domain. For 
these reason, they have been called the “footprints” of turbulence [8]. Based on these measurements, 
designs have been done in the form of Extended Kalman Filter for the spatially discretized Navier-Stokes 
equations [10,11]. 

The area of conducting fluids moving in magnetic fields, even though rich in applications, has only 
been recently considered. Recent results in stabilization of MHD flows make use of nonlinear model 
reduction [4], open-loop control [7], and optimal control [12]. Applications include, for instance, drag 
reduction [18], or mixing enhancement for cooling systems [20]. Some experimental results are available 
as well, showing that control of such flows is technologically feasible [9,18,24]. Mathematical studies of 
controllability have been done, although they do not provide explicit controllers [6,23]. 

For a more detailed presentation of flow control, the reader is referred to the monograph Control of 
Turbulent and Magnetohydrodynamic Channel Flows by Vazquez and Krstic [26]. The book introduces 
new constructive design methods (based on the backstepping technique) for boundary stabilization and 
boundary estimation for several benchmark problems in flow control, with potential applications to 
turbulence control, weather forecasting, and plasma control. It contains the details of methods presented 
here, and includes an introduction on spatial invariance, Fourier series and transforms, the stability of 
infinite-dimensional systems, and the backstepping method for PDEs. It also covers other topics not 
mentioned here, such as well-posedness, H! and H? stability, or Poiseuille flow transfer. 
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73.1 Introduction 


The advent of digital communication networks has created a new subfield of control engineering 
broadly identified as networked control. It addresses the analysis and design of control systems whose 
components are not colocated, and for which the dissemination of information requires a commu- 
nication network. As evidenced in [2], the vast current and future applications of networked con- 
trol systems, and the lack of systematic analysis and design methods, warrants a significant research 
effort. 

Two major challenges in networked control research are the development of analysis and design tools 
that account for the deterioration of performance that digital communication networks cause when they 
are used to connect two or more modules of the feedback loop. This performance degradation is due to the 
typical detrimental features of digital networks such as the finite bit-rate limits, which result from the use 
of data packets carrying a finite number of bits, and erasures or packet losses due to fading, interference 
or congestion. Currently, most research in networked control can be categorized in two types. The first 
focuses on the effects of erasures by assuming that the number of bits carried in each data packet is 
large enough to support the hypothesis that any resulting quantization noise is negligible in light of other 
sources of noise. The second type addresses not only erasures, but it also acknowledges that the effects of 
bit-rate limits and the associated quantization distortion need to be analyzed. In this chapter, we provide 
a brief discussion on the latter, which we complement by citing only a few representative references, on 
a need basis. An informative survey comprising a much richer collection of references, which are up 
to date until 2007, can be found in [3]. An extensive and comprehensive discussion can also be found 
in [31]. 
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73.1.1 Chapter Organization 


This chapter is centered on the stabilizability of unstable finite dimensional linear time invariant (LTI) 
systems via data-rate-limited feedback. The general formulation is given in Section 73.2, while 
Sections 73.3 and 73.4 focus on internal and external stabilization, respectively. 


73.2 Basic Framework 


A prototypical formulation for a networked control system is one where the sensors and the controller 
are not collocated. Here, we assume that these blocks are interconnected by a digital communication 
channel, as shown in Figure 73.1. In our framework the plant, the channel and the remaining blocks 
operate synchronously in discrete time. 

Throughout this chapter, we assume that the digital channels function as random digital links, which 
transmit Ry bits at each time instant k. Here, Ry is a random process taking values in the set {0,1,...,R}, 
and whose statistics reflect the relevant detrimental effects of the underlying network, such as fading 
and congestion [5]. The signal S$; is an ordered R-tuple taking values in {0,1}* and it represents the 
data that is placed for transmission through the channel. The channel operates by transmitting only the 
first Ry bits of the word placed for transmission. Hence, the output of the channel, denoted as Vj, is a 
truncated version of $; where the last R — Ry bits get dropped. This model was adopted in [7] to study 
the effects of channel randomness in the robust stability of networked control systems. The authors of [9] 
have also proposed this type of abstraction in an information theoretic setting. We should, however, note 
that this class of channel models represents a particular case, in light of other more general analog and 
discrete models that represent certain channels more accurately. There is also a significant body of work 
for Gaussian channels, from some of the pioneering work of [12,21] to more recent work addressing 
the difficult problem of networked control over Gaussian channels with memory [15]. A comprehensive 
study for general channels can be found in [8]. 

The two blocks S and K in Figure 73.1 represent the sensing and controller blocks, respectively. The 
controller produces a control signal U; based on causal processing of the output of the channel, while the 
sensing block accepts causal measurements of the output of the plant and uses them to decide when and 
what should be placed for transmission through the channel. The measurement and process exogenous 
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FIGURE 73.1 Depiction of a networked control system. 
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signals are represented by N; and Wj, respectively, while the initial condition of the plant is denoted by 
Xo. Throughout this paper, we assume that {Rx}7°,, and Xo are independent. 


73.3 Internal Stabilization 


We consider that the plant in Figure 73.1 is finite-dimensional LTI, and that it evolves in discrete time. 
The state-space representation of the plant is given as follows: 


Xx41 = AX, +BU;z, k=O (73.1) 
Y, = CX,+DU;z, k>0 (73.2) 


where the initial condition Xo is a random variable. In general, X;, Y;, and U, are vector valued and A, B, 
and C are the matrices of appropriate dimension. 

We refer to the feedback loop in Figure 73.1 as internally stabilizing if the state of the plant (Equa- 
tion 73.1) is stable in the presence of the initial condition Xq. Stability in the presence of external inputs 
is investigated in Section 73.4. 


73.3.1 Deterministic Channel Model 


The simplest case to analyze is when the channel is deterministic, implying that Ry = R holds for all k, 
where R is a predefined positive constant. From the pioneering work in [10,11] it follows that, for a scalar 
LTI plant, internal stabilization via the scheme of Figure 73.1 implies that the following condition must 
hold: 

R> max({0, log, |al} (73.3) 


where a defines the state-space representation of the scalar plant via: 
Xp4) =aX,+BU,z, k>0 (73.4) 


We proceed by giving a simple explanation of Equation 73.3, where, for now, the feedback loop is qualified 
as internally stabilizing if the following holds with probability one: 


[Xk] <v<oo, k>0 (73.5) 


for some positive real B. 
Now consider the following estimate of Xo: 


k-1 
Z=- >) as "BU, k>1 (73.6) 
j=0 


which can be used, via the convolution formula, to express X;, as: 
X, = ak (Xo = zx) SSA (73.7) 
If follows from Equations 73.7 and 73.5 that internal stabilization implies: 
[Xo —Zxl <vlal*, k>1 (73.8) 
Hence, we conclude from Equation 73.8 that Z;. constitutes an estimate of Xo whose error decreases 


exponentially with time. If |a| > 1 holds then the fact that Equation 73.8 holds, regardless of the probability 
distribution of Xo, implies that there exists information about Xo flowing to Z;, at every time step. In 
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particular, with a = 2 the error in Equation 73.8 is reduced by half at every time step, and as a result, there is 
at least one bit of information about Xp being transmitted to Z at every time step. Since Z is constructed 
solely from U, and all information about Xp that it contains must be conveyed through the channel, we 
conclude that for a = 2 it must be true that R > 1. Since for a = 2 it holds that R > 1, it is natural that 
for arbitrary a the necessary condition becomes Equation 73.3. This argument is explained rigorously 
in [10,11], and from it one can conclude that Equation 73.3 is a necessary condition for any arbitrary 
memoryless channel provided that R is replaced with Shannon’s channel capacity, which is a measure 
of the highest information rate attainable by a given channel. As it is shown in [10,11], the inequality 
in Equation 73.3 is also a sufficient condition for the existence of stabilizing S and K. Hence, from 
Equation 73.3, we can clearly conclude that larger |a|, or the more unstable is the system, the larger is the 
required rate R for stabilizability. However, as it was pointed out in [8], if the channel is not deterministic 
then finer notions of stochastic stability are needed, for which the inequality in Equation 73.3, with R 
replaced with capacity, might no longer be a sufficient condition for the existence of stabilizing S and K. 
We will explore an instance of this remark in Section 73.3.2. 

Using an appropriate modal decomposition technique and counting arguments applied to plants of 
arbitrary dimension, the authors of [17] proved that the following is a necessary condition for the existence 
of stabilizing S and K: 


n 
R= J- max{0, log |Aj(A)|} (73.9) 
i=l 

where n is the dimension of the plant and {j;(A)}7_, are the eigenvalues of A. In [17] it is shown 
that the condition (Equation 73.9) is also sufficient if the plant is stabilizable and detectable. Although 
proving this result for the general case is nontrivial, one can gain some insight by noticing that the 
necessity and sufficiency of Equation 73.3, for the stabilization of scalar systems, can be used to derive the 
corresponding condition (Equation 73.9) for multistate plants, for the particular case where A is diagonal, 
or diagonalizable with eigenvalues that are powers of 2. 


73.3.2 Random Channel Model 


In order to access stability in the presence of a random channel, that is, when R, is no longer deterministic, 
we adopt two notions: almost sure stability and moment stability. 
A feedback system is almost surely stable, or a.s.s. for short, if the following holds with probability one: 


|X; S Vass < ©O (73.10) 


Given a positive integer m, an alternative notion is one where a system is mth moment stable if the 
following holds: 
E[|Xx|"]< Vm < 00, k20 (73.11) 


where E[-] denotes statistical expectation with respect to the randomness introduced by the channel and 
the initial condition Xo. 

The authors of [18] have shown that the following is a necessary condition for stabilization in both the 
a.s.s. and the mth moment sense: 


C= Y/ max{0, log |;(A)|} (73.12) 
i=1 
Here C is defined as follows: : 
le 9 Rj 
ef! jimi 2 (73.13) 
HY — k 


in probability 


where lim inf above is assumed to hold in probability. It follows from Equation 73.13 that, for the channel 
model adopted here, C coincides with Shannon’s capacity as defined in the information theory literature. 
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The authors of [19] show that for stabilizable and detectable plants, the condition in Equation 73.12 is 
sufficient for stabilizability, provided that R, is a time-varying deterministic sequence. Here, the assump- 
tion that R; is deterministic guarantees that S knows what is effectively transmitted to K through the 
channel. This predictability underpins the development of algorithms executed at S that can track and 
reproduce any action taken at K, a feature that dramatically simplifies the development of stabilizing 
schemes. This desirable attribute, which we qualify as classical information pattern [1], may not hold for 
random channels. If the channel is random then the information pattern is classical ifthe channel sends an 
acknowledgement signal to S containing R,. Here, the acknowledgement signal effectively indicates how 
many bits are successfully transmitted through the channel from S to K. This type of acknowledgement 
signal may be conveyed via a dedicated link as in Figure 73.1, or it can be signaled by K through the 
plant in certain cases. For channels that are supported on a computer network, one can associate classical 
information patterns with TCP protocols, while UDP protocols are nonclassical. 


73.3.2.1 Classical Information Pattern 


If the channel is memoryless, that is, Ry is white, and the plant is stabilizable and detectable and in 
the presence of acknowledgement signals then the necessary condition in Equation 73.12 may also be 
sufficient for the existence of stabilizing S and K, in the a.s.s. sense. Sufficiency in the absence of process 
noise is proved in [18]. Other results for the multistate case can be found in [4] and references therein, 
where the authors provide explicit stabilizing schemes that rely on the predictability guaranteed by the 
classical information pattern hypothesis. It is interesting to notice that the condition (Equation 73.12) 
may not be sufficient in the presence of process and measurement noises. 

As it is remarked in [8], even if the information pattern of the feedback loop is classical, the necessary 
condition in Equation 73.12 is no longer sufficient to guarantee internal stabilization in the mth moment 
sense. In fact, for scalar plants and memoryless stationary channels, and under a classical information 
pattern, the necessary and sufficient condition for mth moment stabilizability can be cast as follows [7]: 


C > max{0, log, |a|} + B(m) (73.14) 


where B(m) is a positive increasing function of m. The necessary and sufficient condition for stabilization 
for the aforementioned formulation can also be expressed as follows: 


gE |2-" | <1, k=O. (73.15) 


We show next that the condition in Equation 73.15 becomes particularly simple when Rx is governed 
by a Bernoulli process that selects either 0 or R signifying an erasure or a successful transmission, 
respectively. More specifically, if pe denotes the probability of erasure then Equation 73.15 becomes 
a” (e. +2772 — Pe)) < 1. As pointed out in [14], it is interesting to note that as R tends to infinity, the 
aforementioned condition becomes ape < 1, which coincides with the necessary and sufficient condition 
for stabilizability derived in [20] for the packet-drop model. Indeed, one should expect this consistency in 
the limit when R tends to infinity in light of the formulation in [20], which adopts a channel described by 
a Bernoulli-driven device that either causes an erasure (packet drop) or enables the transmission of a real 
number. Here, the transmission of a real number can be viewed as the transmission of a data packet with 
an infinite number of bits. It follows from the aforementioned argument for scalar plants, and by using 
modal decomposition, that a necessary and sufficient condition for the stabilization of stabilizable and 
detectable plants in the mth moment sense with R tending to infinity is given by the following inequality: 


(A) pe <1 (73.16) 


where 0(A) denotes the spectral radius of A. Here, once again, Equation 73.16 coincides with the analogous 
condition derived under the packet drop model [20]. 
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There are other interesting limiting cases of Equation 73.15 studied in [7], such as when m tends 
to zero for which we obtain Equation 73.3 or when m tends to infinity for which we arrive at Ryin > 
max{0, log, |a|}, where Ryin is the minimum value attained by R; with probability one. Similar limiting 
cases were investigated for the multistate case in [14]. 

Existing work on necessary and sufficient conditions for the existence of a stabilizing scheme for a 
plant with multiple states, for the case when Rx is random [14], are not as elegant and simple as their 
counterparts (Equation 73.12) for deterministic channels. A reason for this difficulty is that, as pointed 
out in [7], for multistate plants the blocks S and K have to implement allocation algorithms that dictate 
what information is sent at any particular time instant through the channel. Recent results in [14] provide 
necessary and sufficient conditions for which there is a gap within which the existence of a stabilizing 
scheme cannot be determined. This gap is well characterized in [14], including examples where it is zero. 
The work in [14] also provides explicit algorithms and a thorough statistical analysis. 


Remark 


Some of the necessary and sufficient conditions presented above can be used to determine the stabiliz- 
ability of the plant in the presence of measurement and process noises, provided that they are modified 
to require that the corresponding inequalities hold strictly. 


73.3.2.2 Nonclassical Information Pattern 


In general, if the channel is random and acknowledgement signals from K to S are not available then 
the analysis and design of stabilizing schemes becomes rather involved. This is the case because S does 
not have perfect information about what is successfully transmitted to K. The analysis for this case, 
accompanied by the description of appropriate codes for the stabilization of scalar plants, can be found 
in Section V-A of [8]. Tight necessary and sufficient conditions for multistate plants are, in general, not 
yet known. Clear progress has been made on obtaining necessary and sufficient conditions [16,33] for the 
multistate case provided that the channel is Gaussian additive. 


73.3.3 Decentralized Networked Control 


Decentralized networked control refers to the case in which the plant comprises several dynamically 
coupled subsystems with a subcontroller at each. Here one needs to clearly specify the information 
pattern that dictates how information is disseminated among subcontrollers. An information pattern is 
decentralized if each subcontroller can act directly only on its own subsystem, but has access to only partial 
and possibly degraded information about the state of the other subsystems. The framework described in 
Section 73.2 involves one controller, one plant and one point-to-point channel. Hence, a natural extension 
to the decentralized case is obtained when certain pairs of subcontrollers are “connected” via point-to- 
point channels to exchange measurement as well as other data that is relevant for distributed control. 
The stabilizability via decentralized control schemes must be determined with respect to the properties 
of the plant, the channel, and the information pattern. This problem is unsolved for the general case, but 
significant progress has been made for deterministic channels [26,27,29,30]. 


73.3.4 Notes on Optimality 


Consider the typical optimal control formulation specified by the quintessential cost comprising of 
the expectation of quadratic forms of the state and the control. Under a classical framework, where 
acknowledgments are present, the separation principle holds and it is possible to generate an optimal 
control signal at K based on the optimal state estimate that it constructs from the information transmitted 
through the channel. This is possible because S can use the acknowledgment signals to determine the 
information received by K and use it to indirectly compute the control signal. Hence, there is no incentive 
to use control for signaling and in theory it is possible to determine the optimal K and S. However, 


Control over Digital Networks 73-7 


unfortunately, the optimal estimator at K is, in general, nonlinear and it may be infinite-dimensional and 
time varying. Here, suboptimal solutions might be obtained by propagating an approximate and truncated 
conditional distribution and constructing the corresponding estimator. For the nonclassical case the 
problem can be formulated but, in general, even approximate solutions are elusive. Interesting structural 
results and optimal policies for particular cases have been proposed in [1]. A new characterization of 
nonclassical information structures and its implications for the design of decentralized decision systems 
is discussed in [28]. 


73.4 Input to State Stability 


Consider the following modification of Equation 73.1 so as to the exogenous signals N; and Wx: 
Xp41 =AX,+BU,+W;, k>0, Xo =0 (73.17) 
Y, =CX;,+DU,+Ny, k>0 (73.18) 


The so-called finite gain input to state stability is a mainstay notion in many paradigms of modern 
control, such as Hoo controller design. A feedback loop stabilizes (Equation 73.17) in the finite gain sense 
if and only if there exists a positive real gain y such that the following inequality holds for every possible 
realization of {Ni}P.9> {Wk}Po9 and {Kho ): 


IXIls < nil, W)lle (73.19) 


where V pe (Vidpeg, W pa {Wi }po, and X = {X;}?2, represent realizations of {Nx}729. {Wk}po, and 
{X;}72 > respectively. Here || - ||; and || - ||- are preselected norms, which are appropriately chosen to 
quantify the desired notion of stability. 

Although most results discussed in Section 73.3.2.1 hold in the presence of process and measurement 
noise with a variety of statistical descriptions, they do not address the input-output stability as defined 
in Equation 73.19. In fact, it has been shown in [32] that finite gain stability is not feasible for unstable 
plants that are controlled over a feedback loop comprising one or more bit-rate constrained blocks. 
Hence, in particular, finite gain stability is impossible under the scheme of Figure 73.1. In fact, as it is 
shown in [32], the ratio between any norm of the state and any norm of the external noises becomes 
unbounded in the limits when the exogenous signals are arbitrarily small or arbitrarily large. As pointed 
out in [32], these facts substantially limit the robustness of the resulting networked control systems 
and they also indicate that other notions of stability are required for the performance analysis of these 
systems. 

The following inequality is a relaxation of Equation 73.19: 


|X\ls < B (IGN, W)lle) (73.20) 


where B: Roo > Rso. If the feedback loop is such that B can be selected to be continuous, strictly 
increasing, unbounded and B(0) = 0 holds then we qualify it as input to state stable (ISS), a concept 
discovered and coined in the foundational work in [23]. Using this concept, the authors of [22] show 
that if the plant is stabilizable and detectable and R, is a constant satisfying (Equation 73.12) then there 
exists S and K for which the feedback loop is ISS. An important characteristic of the solution presented 
in [22] is that the algorithms executed at S and K do not have prior knowledge of an upper bound on 
||(N, W)|le. This feature is, in fact, essential to guarantee that the feedback loop is ISS because ISS requires 
that (Equation 73.20) holds for every possible realization of X,N, and W. A key idea in [22] is that of 
designing S as a time-varying quantizer with adaptable resolution, whereby large signal levels are met 
with coarser quantization cells, which can then be made finer when the amplitude of the state lessens. 
This formulation was initially proposed in [24] and its extensions for the ISS case, including robustness 
analysis, are now fully developed in [25]. These results are remarkable in light of the effects that state 
quantization may bring about on the behavior of feedback systems [6]. 
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74.1 Introduction 


This chapter addresses the problem of decentralized control, where multiple controllers have an access 
to different information but need to achieve or optimize a global objective. Most of conventional con- 
trol analysis breaks down when information is decentralized, even in the simplest possible scenarios 
(Witsenhausen, 1968). 

This chapter addresses decentralized control problems in a simple unified framework. This framework 
is introduced in Section 74.2, where we see that a standard controls framework may be utilized, but with 
the addition ofa particular constraint on the controller that needs to be designed. Section 74.3 then briefly 
reviews the parametrization of stabilizing controllers for centralized control, when one does not have this 
decentralization constraint. 

Section 74.4 introduces quadratic invariance—an algebraic condition under which decentralized con- 
trol problems may be cast as convex optimization problems. Section 74.5 looks at particular classes of 
problems to see when this condition holds, and to get some intuition behind when decentralized control 
problems may be tractable. Section 74.6 then discusses the perfectly decentralized control problem, a 
problem which is often of interest yet which typically does not satisfy this condition. Finally, while the 
rest of this chapter focuses on the case where both the system to be controlled and the possible controllers 
are all linear, Section 74.7 discusses some related results for nonlinear control. 
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FIGURE 74.1 Standard feedback control framework. 


74.2 Framework and Setup 


We introduce a unified framework for studying optimal feedback control problems subject to 
decentralized information constraints. 


74.2.1 Standard Framework 


We first review a standard framework for centralized control synthesis. 

Figure 74.1 represents the standard design framework of modern control theory, and is used in many 
other chapters. The signal w represents the vector of exogenous inputs, those the designer has no control 
over, such as wind gusts if one is considering an example in aerospace, and z represents everything 
the designer would like to keep small, which would typically include deviations from a desired state or 
trajectory, or a measure of control effort, for example. The signal y represents the vector of measurements 
that the controller K has access to, and u is the vector of inputs from the controller that feeds back 
into the plant. The plant is subdivided into four blocks that maps w and u into z and y. The block 
which maps the controller input u to the measurements y is simply referred to as G, since it corresponds 
to the plant of classical control analysis, and so that we can later refer to its subdivisions without any 
ambiguity. 

The design objective is to construct a controller K to keep a measure of the size of the mapping from w 
to z, known as the closed-loop map, as small as possible. There are many ways in which one can measure 
the size of a mapping, and thus this basic setup underpins much of modern controls including H2-control 
and Hoo -control. In this framework, a decentralized information structure may be viewed as a constraint 
on the structure of the controller K, as now illustrated by examples. 


74.2.2 Information Constraint 


We now illustrate why, in this framework, decentralization may be simply encapsulated as a constraint 
that the controller lies in a particular subspace. 

The diagram in Figure 74.2 represents three different subsystems, each of which may effect its neighbors, 
and each of which has its own controller, which only has access to measurements coming from its own 
subsystem. In this case, if we look at the system as a whole, we need to design a controller K that can be 


Gy I | Ge I | Gs 
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FIGURE 74.2 Perfectly decentralized control. 
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written as 
uy Ky 0 0 Vi 
u2z)= 0 Ka 0 y2 
U3 0 0 K3 Y3 
——_— 
K 


since each controller input may only depend upon the measurement from its corresponding subsystem. 
In other words, we need to design the best possible K which is block diagonal. The overall problem can 
be viewed as minimizing the size of the closed-loop map subject to the additional constraint that K € S, 
where S is the set of all block diagonal controllers. This concept readily extends to any type of structural 
constraint we may need to impose in formulating an optimal control problem for controller synthesis. 
For instance, if in the above example, each controller was able to share information with its neighbors, 
then we would end up with a constraint set S which is tri-diagonal. In general, the ijth component of the 
controller is held to 0 if the ith controller is unable to see the jth measurement y;. 

If controllers were instead allowed to communicate with each other, but with some delays, this too 
could be reflected in another constraint set S. This situation is represented in Figure 74.3, where the 
controller for a given subsystem i can see the information from another subsystem j after a transmission 
delay of tj. In this case, if we look at the system as a whole, we need to design a controller K that can be 
written as 


uy) Diy, Ku Dr.Ki2 Dr3Kis yi 

uz| =] Dp, Kar Dp, Koo Dr, Ko3 y2 

U3 Di, K31 Dz, K32 Dy, K33 3 
K 


where D;, represents a delay of tj, and Rj represents the part of the controller which we are free to design, 
since each subsystem controller must wait the proscribed amount of time before it can use information 
from each of the other controllers. 

The set S above is called the information constraint, as it captures the information available to various 
parts of the controller. The overarching point is that the objective of decentralized control may be 
considered to be the minimization of a closed-loop map subject to an information constraint K € S. 
The approach is extremely broad, as it seamlessly incorporates any type of decentralization, any control 
objective, and heterogeneous subsystems. It has thus come to be accepted as the canonical problem one 
would like to solve in decentralized control. 


K; Kj 
<—— 
<7 fi 


FIGURE 74.3 Network with delays. 
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74.2.3 Problem Formulation 


The mapping from w to z that we wish to keep small in Figure 74.1, the closed-loop map, can be 
written as f(P, K) = Py, + Pig2K(I — GK )~1P51. The problem that we would like to address may then be 
formulated as 


minimize || f(P, K)|| 
subject to K stabilizes P (74.1) 
KeS. 


The norm (|| - ||) is any appropriate system norm chosen, based on the particular performance objec- 
tives, which could be the H2-norm or Ho -norm as described in detail in other chapters. The information 
constraint S is the subspace of admissible controllers which encapsulates the decentralized nature of the 
system, as exemplified above. The stabilization constraint is needed in the most typical case where the 
signals lie in extended spaces and the plant and controller are rational proper systems whose intercon- 
nections may thus be unstable. It may not be necessary, or another technical condition may be necessary 
such as the invertibility of (J — GK), for other spaces of interest, such as Banach spaces with bounded 
linear operators (Rotkowitz and Lall, 2002, 2006a). 


74.3 Stabilizing Controller Parametrization 


If the plant to be controlled is stable, we could use the following change of variables: 
Q=-K(I—GK)"" —» K=-aQu-GQ)" (74.2) 


and then allowing the new parameter Q to be stable is equivalent to the controller K stabilizing the plant 
P, and the set of all achievable closed-loop maps (ignoring the information constraint) is then given as 


{P11 — Pj2QPo1 | Q stable}. (74.3) 


This is generalized by the Youla-Kucera or YJBK parametrization (Youla, Jabr, and Bonjiorno, 1976), 
which gives a similar change of variables for unstable plants such as allowing the new (Youla) parameter 
Q to vary over all stable systems is still equivalent to considering all stabilizing controllers K, and the set 
of all achievable closed-loop maps is then given by 


{T, — T2QT3 | Q stable} (74.4) 


where T), Tz, T3 are other stable systems. 

We see that these parametrizations allow the set of achievable closed-loop maps to be expressed as 
an affine function of a stable parameter, and thus allow our objective function in our main problem 
(Equation 74.1) to be cast as a convex function of that parameter. However, the information constraint 
K € S will typically not be simple to express in the new parameter, and this will ruin the convexity of the 
optimization problem. 


74.4 Quadratic Invariance 


We have seen that we can employ a change of variables that will make our objective convex, but that will 
generally cause the information constraint to no longer be affine. We thus seek to characterize problems 
for which the information constraint may be written as an affine constraint in the Youla parameter, such 
that a convex reformulation of our main problem will result. 

The following property, first introduced in (Rotkowitz and Lall, 2002), provides that characterization. 
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Definition 74.4.1 


The set S is quadratically invariant with respect to G if 
KGK eS forallKeS 


In other words, given any admissible controller K, the composition KGK has to be admissible as well. 
When this condition holds, it follows that a controller being admissible is further equivalent to the linear- 
fractional transformation we encountered earlier lying in the constraint set (Rotkowitz and Lall, 2006a, 
2006b): 


KeS <> K(I-GK) ‘eS (74.5) 
We can see immediately from Equation 74.2 that for the stable case this results in the equivalence of 
enforcing the information constraint on the controller or on the new parameter: 


KeS => QéeS (74.6) 
and it can be shown that when the plant is unstable, another change of variables can be made such that 
this equivalence still holds (Rotkowitz and Lall, 2006b). 

Thus when the information constraint S is quadratically invariant with respect to the plant G, the 
optimal decentralized control problem (Equation 74.1) may be recast as the following: 
minimize ||T, — T,QT3]| 
subject to Q stable (74.7) 
QeS 


which is a convex optimization problem. 


74.5 Examples 


This section looks at particular classes of information constraints to see when this quadratic invariance 
condition holds, to identify those decentralized problems which are amenable to convex synthesis. We 
see that this algebraic condition often has intuitive interpretations for specific classes of problems. 


74.5.1 Structural Constraints 


We first look at structural constraints, or sparsity constraints, where each subcontroller can see the 
measurements from some subsystems but not from others. This structure can be represented with a 
binary matrix K®. For instance, Kee = 1 if the kth control input u, is allowed to be a function of the 
Ith measurement y;, and Kee = Oif it cannot see that measurement. The information constraint S is then 
the set of all controllers which have the structure proscribed by K®™; that is, all of the controllers such 
that none of the subcontrollers use information which they cannot see. 

A binary matrix G> can similarly be used to give the structure of the plant. For instance, Gn =1 
if Gj is nonzero and the ith measurement y; is affected by the jth control input u;, and GPn = 0 if it 
is unaffected by that input. Given this representation of the structure of the plant and the controller 
constraints, we have the following result: 

S is quadratically invariant with respect to G if and only if 


KpGi Kp" — Kp") =0 for alll i,j,k. 1. (74.8) 


Figure 74.4 illustrates this condition. The condition in Equation 74.8 requires that, for arbitrary i, j, 
k, l, if the three blocks on the bottom are all nonzero (or allowed to be chosen nonzero), then the top 
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FIGURE 74.4 Structural quadratic invariance. 


block must be allowed to be nonzero as well. In other words, if there is an indirect connection from a 
measurement to a control input, then there has to be a direct connection as well. 

When this condition is met, the problem is quadratically invariant, and we can recast our optimal 
decentralized control problem as the convex optimization problem in Equation 74.7. 


74.5.2 Symmetry 


We briefly consider the problem of symmetric synthesis. Suppose that we need to design the best symmet- 
ric controller; that is, the best controller such that Kjj = Ky, for all k, J, and that the information constraint 
S is the set of all such symmetric controllers. If the plant is also symmetric; that is, if Gj = Gj for all i,j, 
then KGK is symmetric for any symmetric K. Thus, KGK € S for all K € S, the problem is quadratically 
invariant, and the optimal symmetric control problem may be recast as Equation 74.7. 


74.5.3 Delays 


We nowreturn to the problem of Figure 74.3, where we have multiple nodes/subsystems, each with its own 
controller, and each subsystem i, can see the information from another subsystem j after a transmission 
delay of tj. 

We similarly consider that the inputs to a given subsystem j may affect other subsystems after some 
delay, and denote the amount of time after which it may affect another subsystem i by the propagation 
delay Pi 

The overall problem of controlling such a network with propagation delays, with controllers that may 
communicate with transmission delays, is depicted in Figure 74.5. 


FIGURE 74.5 Network with delays. 
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When this problem is tested for quadratic invariance, one first finds that the following condition is 
necessary and sufficient: 
thi + pig + ti 2 te for all i, j, k, 1. (74.9) 


This is reminiscent of condition (Equation 74.8) for structural constraints, as it similarly requires that 
any direct path from y; to u,; must be at least as fast as any indirect path through the plant. This condition 
can be further reduced to a very simple intuitive condition (Rotkowitz, Cogill, and Lall, 2005), as long as 
we may assume that the transmission delays themselves satisfy the triangle inequality; that is, 


tit tj = tj for all k, i,j. (74.10) 


This is typically a very reasonable assumption, as it states that information is transmitted between nodes 
in the quickest manner available through the network. If the inequality failed for some k,j, one would 
want to reroute the transmissions from j to k along the faster route such that the inequality would then 
hold. 

If the triangle inequality among transmissions does hold, then condition (Equation 74.9), and thus 
quadratic invariance, is reduced to simply 


py = ty for alli,j. (74.11) 


In other words, for any pair of nodes, information needs to be transmitted faster than the dynamics 
propagate. When this simple condition holds, the problem is quadratically invariant, and the optimal 
decentralized control problem may be recast as the convex problem (Equation 74.7). 

This very intuitive result has a counterintuitive complement when one considers computational delays 
as well. Suppose now that the ith controller cannot use a measurement from the jth subsystem until a pure 
transmission delay of tj, representing the time it takes to send the information from one subsystem to 
the other, as well as a computational delay of c;, representing the time it takes to process the information 
once it is received. 

While intuition might suggest that these two quantities would end up being added and then replacing 
the right-hand side of Equation 74.11, if we now assume that the pure transmission delays satisfy the 
triangle inequality, the condition for quadratic invariance becomes 


pytG>ty foralli,j (74.12) 


with the computational delay on the other side of the inequality. 

This shows that, regardless of computational delay, if information can be transmitted faster than 
dynamics propagate, then the optimal decentralized control problem can be reformulated as a convex 
optimization problem. If we consider a problem with multiple aerial vehicles, for example, where dynamics 
between any pair of subsystems will propagate at the speed of sound, this tells us that transmissions just 
have to be faster than that threshold for the optimal control problem to be recast as (Equation 74.7). 

These results have also been extended to spatio-temporal systems (Rotkowitz, Cogill, and Lall, 2010), 
including the special case of spatially invariant systems. 


74.6 Perfectly Decentralized Control 


We now revisit the problem of Figure 74.2, where each controller can use only the measurements from its 
own subsystem, and thus the information constraint is block diagonal. This problem is never quadratically 
invariant, and will never satisfy condition (Equation 74.8), except for the case where the subsystems do 
not affect one another; that is, except for the case where G is block diagonal as well. 

In all other cases where subsystems may have some affect on others, we thus cannot parametrize all 
of the admissible stabilizing controllers in a convex fashion, and cannot cast the optimal decentralized 


74-8 Control System Advanced Methods 


control problem as a convex problem such as in Equation 74.7. However, a Youla parametrization can 
similarly be used, and while Equation 74.6 does not hold, as the information constraint on the controller 
is not equivalent to enforcing it on the Youla parameter, it is instead equivalent to a quadratic equality 
constraint on the parameter (Manousiouthakis, 1993) 


KeS =} W,+W2Q+ QW3 + QW4Q=0 (74.13) 


for stable operators Wj, W2, W3, W4. When returning to the optimal decentralized control problem, 
this equality constraint replaces the final Q € S constraint of Equation 74.7. The problem is no longer 
convex due to the quadratic term, but the overall difficulty is transformed to one well-understood type of 
constraint, for which many methods exist to approximate optimal solutions. 

Other structural constraints, which are neither block diagonal nor quadratically invariant, can be 
similarly parametrized by first converting them to a perfectly decentralized problem (Rotkowitz, 2010). 


74.7 Nonlinear Decentralized Control 


The framework discussed thus far assumes that the operators, both the plant to be controlled and the 
possible controllers that we may design for it, are all linear, and when applicable, time-invariant as well. A 
similar convex parametrization of stabilizing decentralized controllers exists even when the systems are 
possibly nonlinear and possibly time-varying (NLTV) (Rotkowitz, 2006). The condition allowing for the 
parametrization then becomes 


K,( + GK2) eS forall Ki, Kk. €S. 


When the plant is stable, the stabilizing controllers may be parametrized similarly to Equation 74.3 (Desoer 
and Liu, 1982), and when the plant is unstable, the stabilizing controllers may typically be parametrized 
similarly to Equation 74.4 (Anantharam and Desoer, 1984). Similar to quadratic invariance, the above 
condition then yields the equivalence of the controller and the feedback map satisfying the information 
constraint (Equation 74.5), which then gives the equivalence of the controller and the parameter satisfying 
the constraint as in Equation 74.6. The convex parametrization of all causal stabilizing decentralized 
controllers then results, analogous to the linear case with quadratic invariance. 

While this condition may appear quite different from quadratic invariance, they actually both reduce 
to the same conditions when one considers the classes of sparsity constraints or delay constraints, and so 
these results extend to all of the cases covered in Sections 74.5.1 and 74.5.3. 
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75.1 Introduction 


In this chapter, we will adopt the analog erasure model to describe the communication channel present 
inside a control loop. This model, also referred to as the packet erasure or packet loss model, can be 
described as follows [8]. The channel operates in discrete time steps. At every time step, the channel 
accepts as input a finite-dimensional real vector r(k). The value of the output of the channel y(k) is 
chosen according to an erasure process. At every time step, the erasure process assumes either the value 
T or the value R. If the value at time k is T, y(k + 1) = r(k) and a successful transmission is said to have 
occurred. Otherwise, y(k + 1) = > and an erasure event, or a packet loss, is said to have occurred at time 
k. The symbol ¢ denotes that the receiver does not receive any data; however, the receiver is aware that an 
erasure event has occurred at that time. Note that we have assumed that the channel introduces a constant 
delay of one time step. 

The analog erasure model aims to capture data loss due to a communication channel. Due to effects 
such as interference and fading in wireless channels, congestion in shared networks, or even overload 
and interrupts at a microcontroller, various parts of a control loop between the sensor and controller, 
or the controller and actuator, may exhibit information loss. By considering the idealization in which 
every successful transmission results in the communication of a real vector of a bounded dimension, 
such situations can be modeled using an analog erasure representation. While an analog erasure model 
has an infinite capacity in an information theoretic sense, it is often a useful representation for the cases 
when the communication protocols allow for large data packets to be transmitted at every time step. For 
instance, the minimum size of an ethernet data packet is 72 bytes. This is much more space for carrying 
information than usually required inside a control loop. If the data packets allow for transmission of 
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control and sensing data to a high fidelity, the quantization effects are often ignored and an analog 
erasure model is adopted. 
Various descriptions of the erasure process are possible. The following two models are the most popular: 


1. Maximum Allowable Transmit Interval (MATI)-based models: This model (see, e.g., [3]) is 
described using two integer values nj and nz. For an (1,2) model, in any interval of length 
ny, at most nz erasure events can occur. Apart from this constraint, the erasure process is arbitrary. 

2. Stochastic erasure event-based models: In this model (see, e.g., [12]), the erasure process is arandom 
process. The simplest case is when the erasure events are independent and identically distributed 
at different time steps. In such a case, the erasure process is described by the erasure probability 


Pp 2 Prob(y(k) = ), at any time step k. More complicated models when the erasure process can 
be described by, for example, a Markov chain (possibly on the lines of the Gilbert-Eliot channel 
model [4]) can also be considered. 


In this chapter, we will concentrate on the stochastic erasure event-based models. As with any networked 
control system, two questions can be asked. 


1. What is the effect of introducing channels on the structure of the estimators and controllers? 
2. What are the optimal encoders and decoders that transmit the maximum amount of control 
relevant information to achieve the fundamental limits of performance in such systems? 


These questions are answered in the next two sections, respectively. Section 75.4 lists some extensions to 
the simple model considered here for pedagogical ease, and points out some open research directions. 
Some results on Markovian jump linear systems (MJLS) that are used in the chapter are presented in 
Section 75.5. 


75.2 Effect on Estimation and Control Performance 


75.2.1 Estimation 


We begin with the problem of estimating a linear time-invariant process across an analog erasure channel. 
Consider the setup as shown in Figure 75.1. The process with state x(k) € R” evolves as 


x(k + 1) = Ax(k) + w(k), 


where w(k) is the process noise modeled as white, zero mean, Gaussian with covariance Ry > 0. The initial 
condition x(0) is assumed to be drawn from a Gaussian distribution with zero mean and covariance I(0). 
The state is observed by a sensor of the form 


y(k) = Cx(k) + v(k), 


where v(k) € R? is measurement noise modeled by white, zero mean, Gaussian with covariance Ry > 0. 
We assume that the sources of randomness x(0), {w(j)} and {v(j)}, are mutually independent. The sensor 
transmits its measurements to an estimator across an analog erasure channel with erasure probability p 
at every time step. The pair (A, C) is assumed to be observable. 

The estimator receives those measurements that are transmitted successfully across the channel. It has 
access to the constant matrices A, C, Ry, Ry and T1(0). The estimator aims at generating the minimum 


Process }————>| Sensor }#———» Channel }+———> Estimator 


FIGURE 75.1 Basic setup of the estimation problem. 
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mean-squared error (MMSE) estimate x(k) of the process state x(k) based on the information it has 
access to at time k. If no erasure events were to occur, the optimal estimate is given by the Kalman filter, 
and the estimate error covariance evolves according to a discrete Riccati recursion. In particular, for 
the assumptions as made above, the error covariance is stable in the sense of being bounded as time k 
increases. We wish to extend this analysis to the case when erasure events occur with a probability p at 
every time step. 

Since the event of a packet drop is known to the estimator, the problem is equivalent to estimation of 
the following Markov jump linear system with the jumps occurring according to a Bernoulli process: 


x(k+ 1) = Ax(k) + w(k) 
yk) = Cyyx(k) + v(k), 


where r(k) is the Markov state such that r(k) = 1 with probability 1 — p and r(k) = 2 otherwise. Moreover, 
C, = C and C, = 0. We can thus utilize the standard results from MJLS theory”. In particular, from 
Corollary 75.2 we obtain that the optimal estimator for such a system is provided by a time-varying 
Kalman filter. Due to probabilistic erasure events, the estimate error covariance I1(k) evolves as a random 
variable. An upper bound of the expected estimate error covariance E[I1(k)] is provided by the quantity 
V(k) that evolves as 


V(k+1) =AV(kK)AT + Ry —(1— p)AV(k)C? (cvact ae Ry) CV(k)A’, 


with the initial condition V(0) = (0). A sufficient condition for stabilizability can also be obtained 
through Corollary 75.2. We can also express the condition as a Linear Matrix Inequality (LMI) [14]. The 
system is stabilizable if there exists a matrix X > 0 and a matrix K such that 


X J(1—p)(XA+KC) ./pXA 
JU — p(AtX +C'K") X 0} >0. 
Jpalx 0 x 


For our problem, we can also derive a lower bound on E[I1(k)] as follows. At time step k, the error 
covariance I1(k) is lower bounded by Ry if a packet is received by the estimator, and is equal to AT] 
(k—1)A? + Ry ifa packet is not received. Thus, E[TI(k)] is lower bounded by S(k) which evolves as 


S(k+ 1) =p (AS(K)AT +Ry) + (1 —p)Rw = PASAT + Ry. 


This is a discrete algebraic Lyapunov recursion. By considering the convergence properties of the recur- 
sion, we obtain that a necessary and sufficient condition for stability of S(k) (and, thus, a necessary 
condition for stability of E[II(k)]) is given by po(A)? < 1, where (A) is the spectral radius of A. 


75.2.2 Control 


We now move on to the control problem considered, for example, in [9,14]. To begin with, consider the 
setup shown in Figure 75.2 that has only one channel in the control loop, present between the sensor 
and the controller. Such a situation can arise, for example, when the controller is colocated with the 
process and the sensor is remote, or if the controller has access to large transmission power. The linear 


* A brief review of such results is provided in Section 75.5. 
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Channel 


Estimator/controller |g y 


FIGURE 75.2 Basic setup of the control problem with a single channel present between the sensor and the controller. 


time-invariant process now evolves as 
x(k +1) = Ax(k) + Bu(k) + w(k), 


where the additional variable u(k) € R” is the control input chosen to minimize the cost 


K 
Weeee ps (x7 Qe) a uT (k)Ru(k)) 4exl(K + 1)P(K + x(K + »| 
k=1 


where the expectation at time k is taken with respect to the future values of the packet erasure events, the 
initial condition, and the measurement and process noises. Further, the matrices P(K + 1), Q and R are 
all assumed to be positive definite. The pair (A, B) is assumed to be controllable. 

We can utilize the Markov state representation to solve the Linear Quadratic Gaussian (LQG) problem 
as well. The system can again be written as a Markov jump linear system of the form 


x(k + 1) = Ax(k) + Bu(k) + w(k), 
WK) = Crgyx(k) + v(k); 


where r(k) is the Markov state such that r(k) = 1 with probability 1 — p and r(k) = 2 otherwise. Moreover, 
C; = Cand C) = 0. Thus, we can utilize the separation principle from Section 75.5.3 to obtain the optimal 
controller as the combination of the LQ optimal control, with the MMSE estimate of the state used in place 
of the state value. Note that the MMSE estimate can be calculated as in Section 75.2.1. Moreover, since 
neither the matrix A nor the matrix B depend on the Markov state, the LQ optimal control corresponds 
to the control input for the system 

x(k + 1) = Ax(k) + Bu(k), 


that minimizes the cost function 


K 
a bs (xT do Qx(k) + uT (QRu(k)) +27 (K + PK + Dx(K + o| ; 


k=1 


when the controller has full state information, that is, to calculate the input u(k) at time k, the state 
x(0),x(1),...,x(k) is available. 

We can also consider the case of two channels being present. Consider the system setup shown in 
Figure 75.3, with the process and the sensor as described above. In addition to the sensor-controller 
channel, there is an additional channel between the controller and the actuator. Assume that the erasures 
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FIGURE 75.3 Setup of the control problem with two channels. 


on the controller-actuator channel occur in an iid. fashion, with probability of erasure q at any time 
step. Moreover, the erasure events on this channel are independent of all other random variables in the 
system. 

In this case, it is also important to specify the action that the actuator takes when it does not receive 
a packet. The action depends on the amount of processing, memory, and information about the process 
that is assumed to be available at the actuator. We consider the simplest choice, which is to apply zero 
control input if no packet was received. Other actions by the actuator can be treated in a similar fashion. 
For this case, the Markov jump linear system representation of the system is now given by 


x(k +1) = Ax(k) + Byyyu(k) + w(k) 
yk) = Crayx(k) + v(k), 


where r(k) is the Markov state that can take values 1, 2, 3, and 4 with probabilities (1 — p)(1— q), 
p(l—q), (1—p)q, and pq respectively. The system matrices are given by B; = By = B, B3 = By = 0, 
C, = C3 = C and Cy = C4 = 0. The solution of the LQG problem requires one additional assumption. 
The separation result in Theorem 75.6 assumes that the controller at time k has knowledge of the control 
inputs that were previously applied till time k — 1. Indeed, if the past control inputs are not available at 
the controller, then the control will have a dual effect. For our problem, this implies that the controller 
must know whether or not the transmission over the controller actuator channel has been successful. To 
provide this information to the controller, we will assume a perfect acknowledgment from the actuator 
to the controller for any data packet received by the actuator. This is often called the TCP-like case. 
The case when acknowledgements are not available is termed as the UDP-like case. For the UDP-like 
case, the separation principle does not hold and the form of the optimal controller is unknown, in 
general. 

Note that since the controller can detect any packet drops on the sensor-controller channel, and 
receives acknowledgments about transmissions over the controller—actuator channel, the controller has 
access to the Markov states r(0),...,r(k — 1) at time k, but not r(k). In particular, at time step k, the 
controller does not know the value of B,(x). While in general Theorem 75.6 requires knowledge of the 
current Markov state at the controller, in this particular case, the problem is still solvable. To see this, 
assume that the controller uses the value B,(x) = B to calculate the optimal control input. If indeed r%, = 1 
or 2, the actuator successfully receives this packet and the control input is optimal. If 7, = 3 or 4, the 
optimal control input should be zero since for those states B,(,) = 0. However, at these time steps, the 
transmission by the controller is not successful and the actuator applies zero as the control input. Thus, 
once again, the optimal control input is applied. Thus, we see that the LQG problem can be solved for this 
case using Theorem 75.6. We can also identify stability conditions and performance bounds by utilizing 
the MJLS theory. 
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75.3 Optimal Coding 


We now turn to the more general question of identifying fundamental limits on the performance of a 
system being controlled across an analog erasure channel, and the design of encoders and decoders to 
achieve such limits as discussed, for example, in [7]. To proceed, we must define the class of encoders 
that we will consider. The information theoretic capacity of an analog erasure channel is infinite. Thus, 
the only constraints we impose on the encoder are that the transmitted vector is some causal (possibly 
time-varying) function of the measurements available to the encoder until time k and that the dimension 
of the vector is finite. The encoder is collocated with the sensor, while the decoder is located at the 
estimator/controller. We will sometimes refer to the encoder as an encoding algorithm. 

We begin by considering the system setup in Figure 75.2 and the associated assumptions about the pro- 
cess, the sensor and the cost function in Section 75.2.2. However, instead of transmitting measurements 
y(k) at every time step k, the sensor can now calculate a vector s(k) = f(k, {y( D¥ Lo) and transmit it. Note 
that we have not assumed that the encoder has access to any acknowledgments from the decoder about 
which transmissions have been successful. However, we will show that the presence of such acknowledge- 
ments does not improve the optimal performance achievable by a suitable encoder. 

Denote by Z(k) the information set that the decoder can utilize to calculate the control at time k. As 
an example, if no erasures were happening, Z(k) = {y(0), y(1),...,y(k — 1)}. More generally, given any 
packet erasure pattern, we can define a time stamp f,(k) at every time step k such that the erasures did not 
allow any information transmitted by the encoder after time t,(k) to reach the decoder. Without loss of 
generality, we can restrict attention to information-set feedback controllers. For a given information set, 
L(.) denote the minimal value of the cost JL.gg that can be achieved with the optimal controller design by 
Jigc(Z), and the smallest sigma algebra generated by the information set as I(.). If two information sets 
I'(.) and I?(.) are such that I! (k) C I?(k), we have Jigc(Z’) < Tigc(Z'). 

Consider an algorithm A; in which at every time step k, the sensor transmits all measurements 
y(0), y(1),...,¥(k) to the decoder. Note that this algorithm is not a valid encoding algorithm since 
the dimension of the transmitted vector is not bounded, as k increases. However, with this algo- 
rithm, for any drop sequence, the decoder has access to an information set of the form Z™*(k) = 
{y(0), y(1),...,(ts(k))}, where t;(k) < k — 1 is the time stamp defined above. This is the maximal infor- 
mation set that the decoder can have access to with any algorithm in the sense that I(k) C I™**(k), for any 
other algorithm that yields the information set Z(k). Thus, one way to achieve the optimal value of the 
cost function is to utilize an algorithm that makes Z™*(k) available to the sensor at every time k along 
with a controller that optimally utilizes this set. Further, one such encoder algorithm is A;. However, as 
discussed above, A; is not a valid encoding algorithm. Surprisingly, as shown below, we can achieve the 
same performance with an algorithm that transmits a vector with finite dimension. 

We begin with the following separation principle when the decoder has access to the maximal informa- 
tion set. Denote by &(k|B(k)) the MMSE estimate of the random variable a(k) based on the information 


B(k). 


Theorem 75.1: Separation Principle with Maximal Information Set 


Consider the control problem as defined above, when the decoder has access to the maximal information set 
I™**(k) at every time step. Then, the optimal control input is given by 


uk) = fig (KIE™™E), (UDI) 


where urq(k) is the optimal LQ control law. 
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The proof of this result is similar to the standard separation principle (see, e.g., [10, Chapter 9]) and 
is omitted here. For our setting, the importance of this result lies in the fact that it recognizes that 
ULQ (kIZ™*(K), (uD}ic ) (or, in turn, Xz (kIZ™*(K), tupyed)) is a sufficient statistic to calculate 
the control input that achieves the minimum possible cost for any encoding algorithm. Utilizing the fact 
that the optimal MMSE estimate of x(k) is linear in the effects of the maximal information set and the 
previous control inputs, we can identify the quantity that the encoder should transmit that depends only 
on the measurements. We have the following result. 


Theorem 75.2: Separation of the Effect of the Control Inputs 


The quantity XQ (kIZ™=*(h), (u(DHEE) can be calculated as 


Brg (KIZ™*(K), (uHEE) = Ere (KIZ™* (0) + YOR), 


where XQ (k|Z™*(k)) depends only on I™*(k) but not on the control inputs and wy(k) depends only on 
the control inputs {u({) Hoo: Further both xrq (k|Z™*(k)) and p(k) can be calculated recursively. 


Proof. The proof follows readily from noting that x, (kIZ™(K), {u( ey) can be obtained from the 


Kalman filter which is affine in both measurements and control inputs. We can identify 


XQ (kIZ™*(k)) = AKO -1X(¢,(k) + 1] t(k)), 
k—t,(k)—2 
Wh) = AO GE) + + S> A‘Bu(k—-i-1), 


i=0 


where x(j + 1|j) evolves as 


= M~*(jlj— Xj — 1) + CTR, "y()), 
M(j\j—1) =AM(Gj—1]j— DAT + Ry, 
X(jlj— 1) = AX(j—-1lj—-D), 


with the initial conditions x(0| — 1) = 0 and M(0| — 1) = T1(0), and Wj) evolves as 


Wj) = Buj —1) + PG — WG —D, 
Tj) =AM "(jf — 1 —)M(Gj- 11-2), 


with the initial condition (0) =0. | 


Now consider the following algorithm Az. At every time step k, the encoder calculates and transmits 
the quantity x(k|k) using the algorithm in the above proof. The decoder calculates the quantity (k). 
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If the transmission is successful, the decoder calculates 
R19 (k +1)Z™*(k +1), {u(D}fLo) = kr (k + 1IZ™*(k + 1) + W(k) 
= Ax(k|k) + W(k). 
If the transmission is unsuccessful, the decoder calculates 
B19 (k-+1IZ™™(k-+ 1), (u({)}fe9) = AK P19 (k-+ UZ (i(k) + 1)) + WER), 


where the quantity x,Q(k + 1|Z™*(t;(k) + 1)) is stored in the memory from the last successful transmis- 
sion (note that only the last successful transmission needs to be stored). Using the Theorems 75.1 and 75.2 
clearly allows us to state the following result. 


Theorem 75.3: Optimality of the Algorithm A 


Algorithm A is optimal in the sense that it allows the controller to calculate the control input u(k) that 
minimizes JLQG. 


Proof. At every time step, the algorithm Az makes Rra(k+ 1Z™*(k +1), {u({)}K) available to the 
controller. Thus, the controller can calculate the same control input as with the algorithm A; which 
together with an LQ controller yields the minimum value of Jigga. | 


Note that the optimal algorithm is nonlinear (in particular, it is a switched linear system). This is not 
unexpected, in view of the nonclassical information pattern in the problem. 


Remarks 


¢ Boundedness of the Transmitted Quantity: It should be emphasized that the quantity x(k|k) that 
the encoder transmits is not the estimate of x(k) (or the state of some hypothetical open-loop 
process) based only on the measurements y(0),...,y(k). In particular, under the constraint on 
the erasure probability that we derive later, the state x(k) is stable and hence the measurements 
y(k) are bounded. Thus, the quantity x(k|k) is bounded. This can also be seen from the recursive 
filter used in the proof of Theorem 75.2. If the closed-loop system x(k) is unstable due to high 
erasure probabilities, x(k|k) would, of course, not be bounded. However, the optimality result 
implies that the system cannot be stabilized by transmitting any other bounded quantity (such as 
measurements). 

¢ Optimality for any Erasure Pattern and the “Washing Away” Effect: The optimality of the algorithm 
required no assumption about the erasure statistics. The optimality result holds for an arbitrary 
erasure sequence, and at every time step (not merely in an average sense). Moreover, any successful 
transmission “washes away” the effect of the previous erasures in the sense that it ensures that the 
control input is identical to the case as if all previous transmissions were successful. 

¢ Presence of Delays: We assumed that the communication channel introduces a constant delay of one 
time step. However, the same algorithm continues to remain optimal even if the channel introduces 
larger (or even time-varying) delays, as long as there is the provision of a time stamp from the 
encoder regarding the time it transmits any vector. The decoder uses the packet it receives at any 
time step only if it was transmitted later than the quantity it has stored from the previous time steps. 
If this is not true due to packet reordering, the decoder continues to use the quantity stored from 
previous time steps. Further, if the delays are finite, the stability conditions derived below remain 
unchanged. Infinite delays are equivalent to packet erasures, and can be handled by using the same 
framework. 
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Stability and Performance 


Both the stability and performance of the system with this optimal coding algorithm in place can be 
analyzed by assuming specific models for the erasure process. For pedagogical ease, we adopt the iid. 
erasure model, with an erasure occurring at any time step with probability p. Due to the separation 
principle, to obtain the stability conditions, we need to consider the conditions under which the LQ 
control cost for the system, and the covariance of the estimation error between the state of the process 
x(k) and the estimate at the controller x(k) remain bounded, as time k increases. Under the controllability 
and observability assumptions, the LQ cost remains bounded, if the control value does. Define the 
estimation error and its covariance as 


e(k) = x(k) — x(k), 
P(k) =E [ere] 


where the expectation is taken with respect to the process and measurement noises, and the initial 
condition (but not the erasure process). Due to the “washing away” effect of the algorithm, the error of 
the estimate at the decoder evolves as 


Aiea = ee 1) noerasure 


Ae(k) erasure event, 


where e(k) is the error between x(k) and the estimate of x(k) given all control inputs {u( DY and 


measurements {y( Dre: Thus, the error covariance evolves as 


P(k-+1)= M(k+ : ne probabultty l-p 
AP(k)A* +Ry with probability p, 


where M(k) is the covariance of the error e(k). Thus, we obtain 
E[P(k+1)] = (1—p)M(k+ 1) + pRw + pAE[P(K)IA', 


where the extra expectation for the error covariance is taken over the erasure process in the channel. Since 
the system is observable, M(k) converges geometrically to a steady-state value M*. Thus, the necessary 
and sufficient condition for the convergence of the above discrete algebraic Lyapunov recursion is 


pe(A)? <1, 


where p(A) is the spectral radius of A. Due to the optimality of the algorithm considered above, this 
condition is necessary for stability of the system with any causal encoding algorithm. In particular, for the 
strategy of simply transmitting the latest measurement from the sensor as considered in Section 75.2, this 
condition turns out to be necessary for stability (though not sufficient for a general process model). For 
achieving stability with this condition, we require an encoding strategy, such as the recursive algorithm 
provided above. 

This analysis can be generalized to more general erasure models. For example, for a Gilbert-Eliot type 
channel model, the necessary and sufficient condition for stability is given by 


qoop(A)* <1, 


where goo is the conditional probability of an erasure event at time k+ 1, provided an erasure occurs 
at time k. In addition, by calculating the terms E[P(k)] and the LQ control cost of the system with full 
state information, the performance Jrqg can also be calculated through the separation principle proved 
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above. The value of the cost function thus achieved provides a lower bound to the value of the cost 
function achievable using any other encoding or control algorithm, for the same probability of erasure. 
An alternative viewpoint is to consider the encoding algorithm above as a means for transmitting data with 
lesser frequency to achieve the same level of performance than, for example, transmitting measurements 
to the controller. 


Higher-Order Moments 


It can be seen that the treatment above can be extended to consider the stability of higher-order moments 
of the estimation error, or the state value. In fact, the entire steady-state probability distribution function 
of the estimation error can be calculated. 


75.4 Extensions and Open Questions 


The above framework was explained for a very simple setup of an LQG problem. It is natural to consider its 
generalization to other models by removing various assumptions. We consider some of these assumptions 
below. We also point out some of the open questions. 


¢ Channel between the Controller and the Actuator: The encoding algorithm presented above con- 
tinues to remain optimal when a channel is present between the controller and the actuator (as 
considered in Figure 75.3) as long as there is a provision for acknowledgment from the actuator 
to the controller for any successful transmission, and the protocol that the actuator follows in case 
of an erasure is known at the controller. This is because these two assumptions are enough for the 
separation principle to hold. If no such acknowledgment is available, the control input begins to 
have a dual effect and the optimal algorithm is still unknown. Moreover, the problem of designing 
the optimal encoder for the controller-actuator channel can also be considered. This design will 
intimately depend on the information that is assumed to be known at the actuator (e.g., the cost 
function, the system matrices, etc.). Algorithms that optimize the cost function for such information 
sets are largely unknown. A simpler version of the problem would involve either 
e Analyzing the stability and performance gains for given encoding and decoding algorithms 
employed by the controller and the actuator, respectively. 
e Considering algorithms that are stability optimal, in the sense of designing recursive algo- 
rithms that achieve the largest stability region for any possible causal encoding algorithm. 
Both these directions have seen research activity. For the first direction, algorithms typically involve 
transmitting some future control inputs at every time step, or the actuator using some linear 
combination of past control inputs if an erasure occurs. The second direction has identified the 
stability conditions that are necessary for any causal algorithm. Moreover, recursive designs that 
can achieve stability when these conditions are satisfied have also been identified. Surprisingly, the 
design is in the form of a universal actuator that does not require access to the model of the plant. 
Even if such knowledge were available, the stability conditions do not change. Thus, the design is 
stability optimal. 
¢ Presence of a Communication Network: So far we have concentrated on the case when the sensor and 
the controller are connected using a single communication channel. A typical scenario, particularly 
in a wireless context, would instead involve a communication network with multiple such channels. 
If no encoding algorithm is implemented, and every node in the network (including the sensor) 
transmits simply the measurements, the network can be replaced by a giant erasure channel with 
the equivalent erasure probability being some measure of the reliability of the network. The analysis 
in Section 75.2 carries over to this case; however, the performance degrades rapidly as the network 
size increases. If encoding is permitted, such an equivalence breaks down. The optimal algorithm 
is an extension of the single channel case, and is provided in [6]. The stability and performance 
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calculations are considerably more involved. However, the stability condition has an interesting 
interpretation in terms of the capacity for fluid networks. The necessary and sufficient condition 
for stability can be expressed as the inequality 


Pmax-cutp(A)” < 1, 


where Pmax-cut is the max-cut probability calculated in a manner similar to the min-cut capacity of 
fluid networks. We construct cut-sets by dividing the nodes in the network into two sets with one 
set containing the sensor, and the other containing the controller. For each cut-set, we calculate 
the cut-set erasure probability by multiplying the erasure probabilities of all the channels from 
the set containing the sensor to the set containing the controller. The maximum such cut-set 
erasure probability (over all possible cut-sets) denotes the max-cut probability of the network. The 
improvement in the performance and stability region of the system by using the encoding algorithm 
increases drastically with the size and the complexity of the network. 

¢ Multiple Sensors: Another direction in which the above framework can be extended is to consider 
multiple sensors observing the same process. As with the case with one sensor, one can identify 
the necessary stability conditions and a lower bound for the achievable cost function with any 
causal coding algorithm. These stability conditions are also sufficient and recursive algorithms for 
achieving stability when these conditions are satisfied have been identified. These conditions are a 
natural extension of the stability conditions for the single sensor case. As an example, for the case 
of two sensors described by sensing matrices C; and C2 that transmit data to the controller across 
erasure channels for which erasure events are i.i.d. with probabilities p; and p2 respectively, the 
stability conditions are given by the set 


p2e(Ai)? <1, pip(A2)* <1, pip2p(A)* < 1, 


where p(A;) denotes the spectral radius of the unobservable part of the system matrix A, when the 
pair (A, C;) is represented in the observability cano + nical form. However, the problem of identi- 
fying distributed encoding algorithms to be followed at each sensor for achieving the lower bounds 
on the achieved cost function remains largely open. This problem is related to the track fusion 
problem that considers identifying algorithms for optimal fusion of information from multiple 
sensors that interact intermittently (e.g., see [1]). That transmitting estimates based on local data 
from each sensor is not optimal is long known. While algorithms that achieve a performance close 
to the lower bound of the cost function have been identified, a complete solution is not available. 

¢ Inclusion of More Communication Effects: Our discussion has focussed on modeling the loss of 
data transmitted over the channel. In our discussion of the optimal encoding algorithms, we also 
briefly considered the possibility of data being delayed or received out of order. An important 
direction for future work is to consider other effects due to communication channels. Both from 
a theoretical perspective, and for many applications such as underwater systems, an important 
effect is to impose a limit on the number of bits that can be communicated for every successful 
transmission. Some recent work [11,13] has considered the analog digital channel in which the 
channel supports n bits per time step and transmits them with a certain probability p at every time 
step. Stability conditions for such a channel have been identified and are a natural combination 
of the stability conditions for the analog erasure channel above and the ones for a noiseless digital 
channel, as considered elsewhere in the book. The performance of optimal encoding algorithms 
and the optimal performance that is achievable remain unknown. Another channel effect that has 
largely been ignored is the addition of channel noise to the data received successfully. 

¢ More General Performance Criteria: Our treatment focussed ona particular performance measure—a 
quadratic cost, and the stability notions emanating from that measure. Other cost functions may 
be relevant in applications. Thus the cost function may be related to target tracking, measures such 
as Hy or Hoo [15], or some combination of communication and control costs. The analysis and 
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optimal encoding algorithms for such measures are expected to differ significantly. An an example, 
for target tracking, the properties of the reference signal that needs to be tracked can be expected to 
play a significant role. Similarly, for Hoo related costs, the sufficient statistic, and hence the encoding 
algorithms to transmit it, may be vastly different than the LQG case. Finally, a distributed control 
problem with multiple processes, sensors, and actuators is a natural direction to consider. 

« More General Plant Dynamics: The final direction is to consider plant dynamics that are more 
general than the linear model that we have considered. Moving to models such as jump linear 
systems, hybrid systems, and general nonlinear systems will provide new challenges and results. As 
an example, for nonlinear plants, concepts such as spectral radius no longer hold. Thus, the analysis 
techniques are likely to be different and measures such as Lyapunov exponents and the Lipschitz 
constant for the dynamics will likely become important. 


75.5 Some Results on Markovian Jump Linear Systems 


We present a short overview of Markov jump linear systems. A more thorough and complete treatment 
is given in [2]. Consider a discrete-time discrete-state Markov process with state r(k) € {1,2,--- ,m} at 
time k. Denote the transition probability Prob(r(k + 1) = j|r(k) = i) by qj, and the resultant transition 
probability matrix by Q. Also denote 


Prob(r(k) = j) = 1;(k), 


with 1;(0) as given. The evolution of MJLS, denoted by S; for future reference, can be described by the 
following equations: 


x(k + 1) = Arycgyx(k) + Brgy ulk) + Fg wk), 


(75.1) 
y(k) = Crayx(k) + Gray v(k), 


where w(k) is zero mean white Gaussian noise with covariance Ry, v(k) is zero mean white Gaussian noise 
with covariance R, and the notation X,(,) implies that the matrix X € {X), X2,...,Xm} with the matrix Xj 
being chosen when r(k) = i. The initial state x(0) is assumed to be a zero mean Gaussian random variable 
with variance I1(0). For simplicity, we will consider F,4) = G;(4) =I for all values of r(k) in the sequel. 
We also assume that x(0), {w(k)}, {v(k)} and {r(k)} are mutually independent. 


75.5.1 LQ Control 


The Linear Quadratic Regulator (LQR) problem for the system Sj is posed by assuming that the noises 
w(k) and v(k) are not present. Moreover, the matrix C,(4) = I for all choices of the state r(k). The problem 
aims at designing the control input u(k) to minimize the finite horizon cost function 


K 
Tar = > (Epc ga, [27 OQ) +H? (Ru(K)]) +7 (K + DP(K + 1)x(K + 0), 
k=1 


where the expectation at time k is taken with respect to the future values of the Markov state realization, 
and P(K + 1), Qand Rare all assumed to be positive definite. The controller at time k has access to control 
inputs {u( Me state values {x( Dio and the Markov state values {r( DV Lo Moreover, the system is 


de 
said to be stabilizable if the infinite horizon cost function Jo. al limx-s 90 ee is finite. 


The solution to this problem can readily be obtained through dynamic programming arguments. 
The optimal control is given by the following result. 
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Theorem 75.4: 


Consider the LQR problem posed above for the system S}. 


« At time k, if r(k) =i, then the optimal control input is given by 


u(k) = —(R+BPPAk+ 1B) BPP Ak-+ DAix(R), 


where for j =1,2,...,m, 


m -1 
P(k) =) ay (aa? P,(k + 1)Ay — Aj Pr(k + LB; (R+B; Pik + Br) BE Py (k+ A) 
f=1 


and P(K +1)=P(K+1),Vj=1,2,..., m. 

« Assume that the Markov states reach a stationary probability distribution. A sufficient condition for 
stabilizability of the system is that there exist m positive-definite matrices X,, X2,...,Xm and m* 
matrices Ky,1, K,2,.-.,Kijm,Ka13---sKm,m such that for allj =1,2,...,m, 


m 
Xj > Do ay ((AP + KigBP)XWAT +KijBP)? + Q+KiRK). 


i=1 


Note that the sufficient condition can be cast in alternate forms as linear matrix inequalities, which 
can be efficiently solved. We omit such representations. A special case of Markov jump linear systems is 
when the discrete states are chosen independently from one time step to the next. Since this is the case we 
have concentrated on in this chapter, we summarize the results pertaining to this case below. 


Corollary 75.1: 


Consider system S, with the additional assumption that the Markov transition probability matrix is such 
that for all states i and j, qi =i (in other words, the states are chosen independently and identically 
distributed from one time step to the next). Consider the LQR problem posed above for the system 8. 


« At time k, if r(k) =i, then the optimal control input is given by 
-1 
u(k)=—(R+BPP(k+1)Bi) BY P(k+ DAix(h), 
where 


7 T is T mer 
PUK) = oar (Q+APP(K+ 1A: — AT P(R+ DB, (R+B/P(R+ 1B) BY P(R+ IAL J. 
t=1 
« Assume that the Markov states reach a stationary probability distribution. A sufficient condition 


for stabilizability of the system is that there exists a positive-definite matrix X, and m matrices K, 
Ky, ..., Ky such that 


m 
X> Dog (AP + KiBI)XCAT + KiBI)" +Q+KiRK}). 


i=1 
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75.5.2 MMSE Estimation 


The MMSE estimate problem for the system Sj is posed by assuming that the control u,(4) is identically 
zero. The objective is to identify at every time step k an estimate x(k+ 1) of the state x(k+ 1) that 
minimizes the mean-squared error covariance 


Mk +1) = Ejwcpysocp) [K+ 1) — 8+ Yok +) —HE+ DT], 


where the expectation is taken with respect to the process and measurement noises (but not the Markov 
state realization). The estimator at time k has access to observations {y(j/)} 0 and the Markov state values 


{r( DY Lo Moreover, the error covariance is said to be stable if the expected steady-state error covariance 


limps 09 Eup [I1(k)] is bounded, where the expectation is taken with respect to the Markov process. 
j=0 


The estimator at time k has access to the measurements y(0), y(1),...,y(k), and Markov state values 
r(0), r(1),..., r(k). 

Since the estimator has access to the Markov state values till time k, the optimal estimate can be 
calculated through a time-varying Kalman filter. Thus, if at time k, r, = i, the estimate evolves as 


R(k +1) = Aj&(k) + K(k) (y(k) — C:X(k)) , 
where 
T T a 
K(k) = AjTI(K)C; (cindnc, +Ry) 
-1 
M(k +1) = AiTM(K)AP + Ry —AiT(K)CT (CiTMK)CT + Rv) CiTIGO AT. 
The error covariance II(k) is available through the above calculations. However, calculating 


Evy [I1(k)] seems to be intractable. Instead, we present an upper bound to this quantity* that will 
j=0 


also help in obtaining sufficient conditions for the error covariance to be stable. 

The intuition behind obtaining the upper bound is simple. The optimal estimator presented above 
optimally utilizes the information about the Markov states till time k. Consider an alternate estimator 
that at every time step k averages over the values of the Markov states r9,..., 7,1. Such an estimator 
is suboptimal and the error covariance for this estimator forms an upper bound for Eunice [T1(k)]. A 


formal proof using Jensen’s inequality is given in [5, Theorem 5]. We present the statement below while 
omitting the proof. 


Theorem 75.5: 


The term Eps [T1(k)] obtained from the optimal estimator is upper bounded by M(k) = bas Mj(k) 
7=0 


where 


m -1 
My(k) = > ae (® + A;M(k — 1)A? — AyM,(k — 1)CP (R, + C,M,(k— 1)CP ) C:Mi(k — vat) ; 
t=1 


with Mj(0) = T1(0) Vj. Moreover, assume that the Markov states reach a stationary probability distribution. 
A sufficient condition for stabilizability of the system is that there exist m positive-definite matrices X,, 
X>,...,Xm and m? matrices Ky, Ki2,..-, Kim K21..-+»Kinm such that for allj = 1,2,...,m, 


m 
X> > ay (4: + KjjCj)Xi(Ai + KijCj)7 + Rw + KjRvK}) ; 


i=1 


* 


We say that A is upperbounded by B if B — A is positive semi definite. 
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We can once again consider the special case of states being chosen in an independent and identically 
distributed fashion. 


Corollary 75.2: 


Consider the estimation problem posed above for the system S, with the additional assumption that the 

Markov transition probability matrix is such that for all states i and j, qi; = qi (in other words, the states are 

chosen independently and identically distributed from one time step to the next). The term Ep [T1(k)] 
j=0 


obtained from the optimal estimator is upper bounded by M(k), where 


li -1 
M(k)= doar (®. +A,M(k— AT — A,M(k—1)CP (R, +C,M(k— CT ) CM(k— At ) 


t=1 


with M(0) =I1(0). Further, a sufficient condition for stabilizability of the system is that there exists a 
positive-definite matrix X, and m matrices Kj, K2,..., Km such that 


m 
X> ogi ((Ai+KiC)XA +KIC)? + Rw + KIRK?) 


i=1 


75.5.3 LQG Control 


The LQG problem for the system S; aims at designing the control input u(k) to minimize the finite 
horizon cost function 


K 
jee=E pb» (x" GQx(h) $ u (k)Ru(k)) 4x7 (K + 1)P(K + 1)x(K + »| 
k= 


1 


where the expectation at time k is taken with respect to the future values of the Markov state realization, 
and the measurement and process noises. Further, the matrices P(K + 1), Q and R are all assumed to be 


positive definite. The controller at time k has access to control inputs {u( D208 measurements {y( DYLo 
and the Markov state values {r( DYLo The system is said to be stabilizable if the infinite horizon cost 


de 
function Joo a limg-+00 hes is finite. 
The solution to this problem is provided by a separation principle and using Theorems 75.4 and 75.5. 
We have the following result. 


Theorem 75.6: 


Consider the LQG problem for the system S,. At time k, if r(k) = i, then the optimal control input is given 
by 


-1 
iif) = = (R + BIP(k-+ 1)Bi) BI i(k + 1)A:X(k), 
where for P;(k) is calculated as in Theorem 75.4 and x(k) is calculated using a time-varying Kalman filter. 


We can also obtain the conditions for stabilizability of the system by utilizing Theorems 75.4 and 75.5. 
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This chapter reviews a passivity-based approach for the design and analysis of networked nonlinear 
systems. This approach exploits the structure of the network and breaks up the design and analysis proce- 
dures into two levels: At the network level, one represents the components with input/output properties, 
such as passivity and other forms of dissipativity, as abstractions of their complex dynamic models and 
determines which input/output properties guarantee stability and other desirable properties for the given 
network structure. This means that the study of the network does not rely on detailed knowledge of the 
components and does not require the components to be homogeneous. At the component level, one stud- 
ies the individual dynamic models and verifies or assigns appropriate input/output properties without 
relying on further knowledge of the network. 


76-1 
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The question of when a network of dissipative dynamic systems is stable was studied in the early 
Refs. [1,2] and several tests were developed. From today’s standpoint, these studies are encompassed by 
the elegant and unifying framework of Integral Quadratic Constraints [3] discussed elsewhere in this 
handbook. The objectives of this review are 


1. To present recurrent interconnection structures in several modern networks and to show that a 
key input/output property compatible with these structures is passivity. 
2. To illustrate the verification and assignment of passivity properties in these network models. 


The second task is particularly challenging in applications where the equilibrium of the network model 
and other global parameters are not available to the components. 

We illustrate the passivity approach with case studies in communication networks, motion coordina- 
tion of autonomous agents, and biochemical reaction networks. In Section 76.2 we study an intercon- 
nection structure that arises in decentralized resource allocation algorithms in communication networks, 
including congestion control for the Internet discussed in Section 76.2.1 and power control for wireless 
networks discussed in Section 76.2.2. This structure exhibits a symmetry in the coupling of the compo- 
nents, which implies that a passivity property of the update algorithms (for packet rates, transmission 
power levels, etc.) guarantees stability of the network. With this observation as a starting point, we present 
a systematic framework which, rather than giving specific algorithms, prescribes a passivity constraint 
that new classes of algorithms must satisfy. 

In Section 76.3, we show that a similar symmetric-coupling structure arises in motion coordination 
when the information flow between the agents is bidirectional. Passivity-based design for motion coor- 
dination is related to potential function-based schemes, such as those in [4-6]. By making explicit the 
inherent passivity property in these schemes, in Section 76.3.1 we present a systematic framework that 
allows complex, nonlinear agent models and that offers additional design flexibility. The ability of the pas- 
sivity framework to address complex agent dynamics is illustrated with an attitude coordination design 
for rigid body models in Section 3.2. The design flexibility is illustrated with an adaptive redesign in 
Section 76.3.3. 

In Section 76.4, we present a passivity-based analysis of biochemical reaction networks. In Section 
76.4.1 we discuss a cyclic interconnection structure that is common in biological systems and show how 
the passivity-based approach recovers and significantly strengthens the secant criterion [7,8] used by 
mathematical biologists for studying the stability of such networks. After identifying passivity as the core 
of the secant criterion, in Section 76.4.2 we generalize this criterion to other topologies using the concept 
of diagonal stability [9,10]. Finally, in Section 76.4.3 we employ passivity properties to ensure robustness 
against the destabilizing effect of diffusion in spatially distributed models. 

Following [11], we say that a dynamic system 


x=f(x,u), y=h(x,u), (76.1) 


where input u and output y are of the same dimension, is passive if there exists a continuously differentiable 
storage function S(x) > 0 such that 


S = VS(x)f(x,u) < uly. (76.2) 


The Passivity Theorem [12], when adapted to systems in state-space form, states that the negative feedback 
interconnection of two passive blocks as in Figure 76.1a with positive definite storage functions is stable. 
Indeed, the sum of the storage functions of individual blocks serves as a composite Lyapunov function 
for the interconnected system. A further property of passive systems which is particularly useful in this 
article is that postmultiplication of the output y by a matrix and premultiplication of the input u by the 
transpose of the same matrix as in Figure 76.1b maintain passivity with respect to the new input-output 
pair (u, y). 
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FIGURE 76.1 (a) Negative feedback interconnection of two passive systems. (b) Postmultiplication of the output y 


by a matrix and premultiplication of the input u by the transpose of the same matrix maintain passivity with respect 
to the new input-output pair (4, y). 


76.2 Resource Allocation in Communication Networks 


76.2.1 A Unifying Passivity Framework for Internet Congestion Control 


Congestion control algorithms aim to maximize network throughput while ensuring an equitable alloca- 
tion of bandwidth to the users. In a decentralized congestion control scheme, each link increases its packet 
drop or marking probability (interpreted as the “price” of the link) as the transmission rate approaches 
the capacity of the link. Sources then adjust their sending rates based on the aggregate price feedback 
they receive in the form of dropped or marked packets. Reference [13] showed that a passivity-based 
design is particularly suitable for this decentralized feedback structure and developed a unifying design 


methodology. 
To see the interconnection structure of sources and links, consider a network where packets from 
sources i= 1,...,N are routed through links = 1,...,L according to an L x N routing matrix R in 


which the (¢, i) entry is 1 if source i uses link @ and 0 otherwise. Because the transmission rate yg of link @ 
is the sum of the sending rates x; of sources using that link, the vectors of link rates y and source rates x 
are related by 


y= Rx. (76.3) 


Likewise, since the total price feedback q; received by source i is the sum of the prices pg of the links on 
its path, the vectors q and p are related by 


q=R'p. (76.4) 


The resulting feedback loop is depicted in Figure 76.2 where the blocks &; represent decentralized update 
algorithms for source rates x;, and A¢ represent decentralized update algorithms for link prices pg. 

The congestion control problem as formulated by Kelly et al. [14] and studied by numerous other 
authors (see the excellent reviews [15,16]) is to design these algorithms in such a way that the closed-loop 
system is stable and the network equilibrium (x*, y*) solves the optimization problem: 


N 
ne 2 Uj(xi) st. ye <ce, (76.5) 
i= 


where Uj(-) is a concave utility function for source i and cg is the capacity of link 2. The key observation 
in [13] is that postmultiplication by R and premultiplication by R’ in Figure 76.2 preserve passivity 
properties and thus, a design that renders &; passive from input q; — qj to output —(x; — x) and Ag 
passive from input ye — y7 to output pe — p; guarantees stability of the network according to the Passivity 
Theorem. 

The passivity framework streamlines the design process by eliminating the need for a separate network 
stability analysis for every new algorithm. Indeed, numerous existing update rules already satisfy the 
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FIGURE 76.2 The feedback structure arising in decentralized congestion control. R is the routing matrix, X; blocks 
represent the source algorithms for updating sending rates x;, and A; blocks represent the link algorithms for updating 
link prices py. 


desired passivity property, such as Kelly’s primal algorithm: 
& = ki(U;(xi)— qi) pe = he(ye)s (76.6) 


where kj > 0, U;(-) is the derivative of the utility function U;(-), and hg(-) is a penalty function that grows 
rapidly as ye approaches the link capacity cg. The network equilibrium resulting from this algorithm 
approximates the solution of the Kuhn-Tucker optimality conditions for Equation 76.5 and the stability 
of this equilibrium follows from passivity properties established in [13] by exploiting the monotone 
decreasing property of U;(-) and monotone increasing property of he(-). 

As an illustration of the design flexibility offered by the passivity framework [13] presented new classes 
of algorithms, including the following extension of the source control algorithm in Equation 76.6 


&; = Ai; + Bi(U;(xi) — qi), (76.7) 
&j = Ci8i + Di(Uj (xi) — qi), (76.8) 


where the matrices A;, B;, C;, Dj must be selected such that the transfer function H;(s) = C;(sI — Aj)~!B; + 
D; is strictly positive-real (SPR). The benefit of increasing the dynamic order of Equation 76.6 is demon- 
strated in [13] with an example where the SPR filter design adds phase lead to counter time delays. 

A passivity analysis and a dynamic extension was also pursued in [13] for Kelly’s dual algorithm where, 
in contrast to the primal algorithm (Equation 76.6), the source controller is static and the link controller is 
dynamic. In addition, the passivity approach has made stability proofs and systematic Lyapunov function 
constructions possible for primal-dual algorithms where both source and link controllers are dynamic. 
As an illustration, for the following algorithm in [16] 


4 = fi(xi)(U; (xi) — qi), (76.9) 
Pe = gelpe)(ve — ce)s (76.10) 


where fj(-) and ge(-) are positive-valued functions, a Lyapunov function constructed from a sum of storage 


functions is es 
xa HF ee p= Ve 
y= i idx + dp. (76.11) 
Dob he) 2 my gel) 


Lyapunov functions obtained within the passivity framework have also been instrumental in robust 
redesigns against disturbances, time delays, and uncooperative users [17,18]. 


76.2.2 CDMA Power Control Game 


Another important resource allocation problem for communication networks is uplink transmission 
power control in code division multiple access (CDMA) systems. Increased power levels ensure longer 
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transmission distance and higher data transfer rate, but also increase battery consumption and interfer- 
ence to neighboring users. A particularly elegant approach to this problem is a game-theoretic formula- 
tion [19] where the cost function for the ith mobile is 


Ji = Pilpi) — Uilyi(p)), i=1,...,N. (76.12) 


P;(-) is a penalty function for the power level p; and U;(-) is a utility function for the signal-to-interference 
ratio (SIR), given by 

Lhip; 
+ ges MkPk 


where h; is the channel gain between the ith mobile and the base station, L is the spreading gain and o? is 
the noise variance. 
To develop a distributed power update law, [20] employed the logarithmic utility function 


yilp) = (76.13) 


U;(yi) = log(L + yi) (76.14) 


and studied a first-order gradient algorithm which, upon algebraic manipulations, is given by the 
expression 


é Oi i 
j= —M— = —MPi(p) + i> _.. (76.15) 
° Opi 7 + Diy AkPk 
This algorithm is to be implemented by the mobiles with the help of the feedback signal: 
h; 
pe == (76.16) 
o + Dik AePk 


received from the base station. The resulting feedback structure of the network is depicted in Figure 76.3 
where 4; represents the update law (Equation 76.15) for p;, h denotes the column vector of channel gains, 
and Equation 76.16 is represented as a function of y := }~, hyp, in the feedback block. 

The equilibrium p* of Equation 76.15 is unique when P;(-) is strictly convex and coincides with the 
Nash equilibrium for the game defined by the cost function (Equation 76.12). Global stability of this 
equilibrium follows from the Passivity Theorem because premultiplication by h and postmultiplication 
by h? in Figure 76.3 preserve passivity of the D; blocks in the feedforward path from input uj — u* to 
output y; — y;. Likewise, the function (y) is strictly increasing and thus, the feedback block is passive 
from input y — y* to output (y) — o(y*). 

Reference [20] uses this observation as a starting point to develop classes of passive power update laws 
and base-station algorithms that include Equations 76.15 and 76.16 as special cases. It further employs the 


FIGURE 76.3 The feedback structure arising from the distributed power update law (Equation 76.15). The 2; blocks 
represent the update laws p; = —j;P/(pi) + diuj, h denotes the column vector of channel gains, and Equation 76.16 is 
represented as a function of y := }°, hyp, in the feedback block. 
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Lyapunov function obtained from the Passivity Theorem to study robustness of power control algorithms 
against a time-varying channel gain h(t) which is assumed to be constant in the nominal stability analysis. 
A related paper [21] studies a team-optimization approach to power control rather than the game- 
theoretic formulation discussed above and pursues a passivity-based design. 


76.3 Distributed Feedback Design for Motion Coordination 


76.3.1 Passivity-Based Design Procedure for Position Coordination 


We represent a group of agents and their communication structure with a graph that consists of N nodes 
connected by M links. The presence of a link between nodes i and j means that agents i andj have access to 
the relative distance information x; — x;. We assign an orientation to each link and recall that the N x M 
incidence matrix D of the graph is defined as 


+1 if node i is the positive end of link k 
dix = 4—1  ifnode iis the negative end of link k (76.17) 
0 otherwise. 


The assignment of orientation is for analysis only, and the particular choice does not change the results. 
Our objective is to develop distributed feedback laws that obey the information structure defined by this 
graph and that guarantee the following group behaviors: 


P1: The velocity of each node approaches a common velocity vector v(t) prescribed for the group; that is, 
limy5o0(%; — v4(t)) =0,i=1,...,N. 
P2: If nodes i and j are neighbors connected by link k, then the difference variable 


N oe ae , 
so! _ Jxi—+xj  ifiis the positive end of link k 
cuca az dexxe = le —x; ifiis the negative end of link k Voss) 


converges to a prescribed set Ay, k=1,...,M. 


Examples of sets A; include the origin in a rendezvous problem, or a sphere if the positions of agents 
must maintain a prescribed distance in a formation. From Equation 76.18, the concatenated vectors 


xi= [xt see Peele Zi lz? see el (76.19) 


satisfy 
z=(D' @I)x, (76.20) 


where J is an identity matrix with dimension consistent with that of x; and “®” represents the Kronecker 
product. 

To achieve P1 and P2 [22] presented a two-step design procedure: Step 1 is to design an internal 
feedback loop for each node that achieves passivity from an external feedback signal uS*', left to be 
designed in Step 2, to the velocity error y; := x; — v4(t). This design step is depicted with a block diagram 
in Figure 76.4, where H; represents the open-loop dynamic model of agent i and H; represents the passive 
block obtained from the internal feedback design. Step 2 is to design an external feedback law of the form 


M 
uf = — Do dit Wi (zk), (76.21) 
k=1 


where z;’s are the relative distance variables as in Equation 76.18 and the multivariable nonlinearities 
Wx(-) are to be designed. The feedback law (Equation 76.21) is implementable with locally available signals 
because diz # 0 only for links k that are connected to node i. 
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(a) (b) v(t) 


FIGURE 76.4 The task of the internal feedback design is to render the plant (a) passive from the external signal 
ust to the velocity error yj ‘= xj — v(t). With this internal feedback, the dynamics take the form of (b) where the Hi 
block is passive. 


The combination of the internal and external feedback loops result in the interconnected system in 
Figure 76.5, where asymptotic stabilization of the set 


A=[@ly=0, ze (Arx-+- x Au} ORD @D} (76.22) 


is synonymous to achieving objectives P1-P2 above. To accomplish this stabilization task, we exploit 
the interconnection structure in Figure 76.5 where premultiplication by D’ @ I and postmultiplication 
by its transpose D @ I preserve passivity properties of the feedforward path. The input ly @ v4(t) does 
not affect the feedback loop in Figure 76.5 because it lies in the null space of D’ @ I. We design the 
nonlinearities iy; to be passive when cascaded with an integrator as in Figure 76.5 so that the feedforward 
path is passive from input y to output —uext and, thus, the feedback loop is stable from the Passivity 
Theorem. 
To guarantee passivity of the nonlinearity 4(z,) preceded by an integrator, we let 


Wx(Z) = VPx(Z); (76.23) 


where P;,(z,) is a nonnegative and sufficiently smooth function defined on an open set G, in which z, is 
allowed to evolve. To steer z, to A; while keeping it within G;, we construct the function P;.(z;,) to grow 
unbounded as z;, approaches the boundary of Gx, and let P;,.(z;,) and its gradient V P;,.(z,) vanish on the set 
Ax. Using this construction and the passivity properties of the feedback and feedforward paths in Figure 
76.5, [22] proves asymptotic stability of the set A. 

As an illustration, consider the point mass model 


Xj = Uj, (76.24) 


where x; € R? is the position of each mass and u; € R? is the force input. The internal feedback 


uj = —Kj(x; — v4(t)) + 4(t) tu, =K;=K} >0 (76.25) 
a] 
1y® v(t) z w2 W Hext 
L> D®I 
J lwo 
Hy 2 
Yoxt 
< 
Hy 


FIGURE 76.5 A block diagram representation for the interconnection of the subsystems in Figure 76.4 via the 
external feedback (Equation 76.21). 
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and the change of variables y; = x; — v4(t) bring Equation 76.24 to the form 
xj =yi tv7(0), (76.26) 
j= Kip + ut, (76.27) 


ext 


where the y;-subsystem with input u;*' plays the role of the passive block H,; in Figure 76.4. 
To create and stabilize an equilateral triangle formation with unit side lengths while avoiding collisions, 
we let A, be the unit circle, G, = R? — {0}, and let the potential functions be of the form 


\Zl 
Px(Zk) = / ox(s)ds, k=1,2,3, (76.28) 
1 
where ox : Ryo — R is a continuously differentiable, strictly increasing function such that 
o4(1)=0 lim ox(s) = —oo, lim ox (s) = —0o (76.29) 
S—> 00 s>0 
and such that, as |z,| + 0, Px (zx) —> oo. Then, the interaction forces 


1 
We(Ze) = VPr(z%) = ox(l2el) 7 2 2 #0 (76.30) 


guarantee asymptotic stability of the desired formation. In particular, o;(|z,|) creates an attraction force 
when |z,| > 1 and a repulsion force when |z;,| < 1. 


76.3.2 From Point Mass to Rigid-Body Models 


A key advantage of the passivity framework is its ability to address high-order and complex agent dynamics 
by exploiting their inherent passivity properties. As an illustration, we now study a rigid-body model and 
design a controller that achieves identical orientation and synchronous rotation of the agents. This means 
that the objectives P1 and P2 in Section 76.3.1 above must now be modified as 


Al: The angular velocity of each agent converges to the group reference w(t); that is, Jim (1; — w4(t)) = 
00 


0, i=1,...N, where 'w; denotes the angular velocity of agent i in the ith body frame. 
A2: Each agent achieves the same attitude as its neighbors in the limit; that is, the orientation matrices 
R; satisfy lim R} Rj =1,i,j=1,...,N. 
t>oo 


To achieve objectives Al and A2 we first design an internal feedback loop 1; for each agenti = 1,...,N 
that renders the attitude dynamics 


Gj: 'T, ig; + ‘ew; x 'T; cay = Tj (76.31) 
passive from an external input signal 15*' left to be designed, to the angular velocity error 
Aa; :='@;— 04 (t). (76.32) 


One such controller is 
yj ='To4 +04 x'Zj'wj—fido ti, fi>0, (76.33) 


which indeed achieves strict passivity from t§* to Aw; in the error dynamics system: 
G; y 'T, ha; + Aaj; X 'T, ‘(; = —fiAw; + eee (76.34) 


Other designs (possibly using dynamic controllers) that achieve passivity from t$*t to Aw; may also be 
employed in this framework. This design flexibility is illustrated in the next section with an adaptive 
redesign. 
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To achieve objective A2 using only relative attitude information, we parameterize the relative orienta- 
tion matrix 


=k RIR if node i is the positive end of link k 
= Ti (76.35) 


RU Ri if node i is the negative end of link k 


using one of the standard parameterizations of SO(3), such as the unit quaternion representation q* = 


ryt ; ; 
[af gk | , where q* denotes the vector part of the quaternion. We then design an external torque 


feedback of the form 
Se as Ye as (76.36) 
IeNj  peN, 


where N,* denotes the set of links for which node i is the positive end and V, is the set of links for 
which node i is the negative end. To synthesize this external feedback signal, agent i obtains its neighbors’ 
relative attitudes with respect to its own frame, parameterizes them by unit quaternions, and adds up 
their vector parts. 

The closed-loop system resulting from the internal and external feedback laws (Equations 76.33 and 
76.36) is depicted in Figure 76.6, where the J, blocks represent the quaternion kinematics, which possess 
passivity properties from the relative angular velocity a* to the vector component qf of the unit quaternion 
representation [23]. To incorporate the rotation matrices between the body frames, we replace the matrix 
D®1 in Figure 76.5 with a new 3N x 3M rotational incidence matrix D, which consists of the 3 x 3 


sub-blocks: 
7 -I if node i is the positive end of link k 


diz =: 4 (R*)? if node iis the negative end of link k (76.37) 
0 otherwise. 


As shown in [24], stability of the closed-loop system in Figure 76.6 follows from the Passivity Theorem 
because premultiplication by D’ and postmultiplication by D preserve passivity of the feedforward 
path, and because the feedback path is passive by the internal feedback design. Although the foregoing 
arguments are based on the unit quaternion representation of SO(3), they have been generalized in [24] 
to other parameterizations. 


76.3.3 Adaptive Redesign 


Thus far we assumed that the reference velocities v4(t) and w@(t) in objectives P1 and Al above are 
available to each agent in the group. A more realistic situation is when a leader in the group possesses 
or autonomously determines this information, while others have access only to the relative distance and 
relative orientation with respect to their neighbors. Can the agents estimate v4(t) and w4(t) online from 


ji 
d 
1y@ 0) oF _T 6 Jy I D 


Im 


dS 


Aw ext 


Gu 


FIGURE 76.6 A block diagram representation for the network resulting from the attitude coordination scheme 
(Equations 76.33 and 76.36). The concatenated vector w consists of the angular velocities of the agents in their own 
body frames and the rotational incidence matrix D is as defined in Equation 76.37. 
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this relative distance and orientation information? We now present an adaptive redesign from [25] that 
accomplishes this task by relying only on the connectivity of the graph and the passivity properties of the 
interconnected system. 

This adaptive redesign modifies the internal feedback loop to assign the estimate 7; and ‘@; to agent i, 
instead of the unknown references v4(t) and w@(t). To develop update laws for ¥j and 1), we parameterize 
v4(t) and w4(t) as 

V()= Slo, wt) =) vip, (76.38) 


J J 


where ¢/(f) and y/(t) are scalar base functions known to each agent, and 6/ and B/ are vectors available 


only to the leader. Agent i estimates the unknown 6/ and B/ by 0) and ! , and reconstructs ?;(t) and ‘'@;(t) 
from ; ; 
H(t) = Y> 64(H6}, 1O,(t) = D> vB}. (76.39) 


j j=l 


The update laws proposed in [25] are of the form 
6; = Aj(®(t) @ Dus, 6; =ATHepu (76.40) 


where A; = Ar >0and A; = AT > 0 are adaptation gain matrices, ®(t), I(t), 6;, Bi denote concate- 


nations of /(t), y/(t), 0) 5 6! respectively, and u£*t and t{*' are the external force and torque feedback 
laws. The adaptation of 6; and B; continues until the group reaches the desired position and orientation 
configuration, in which case the external force and torque feedback signals employed in Equation 76.40 
vanish. 

Reference [25] proves that this adaptive scheme indeed stabilizes the desired configuration by exploiting 
the passivity of the adaptation algorithm, which is depicted in Figure 76.7 for position coordination as 
an additional module in the existing passive feedback loop. Parameter convergence is established under 
additional conditions. One situation in which parameter convergence is guaranteed is when the reference 
velocity v4 is constant and the graph is connected, in which case the proof in [25] makes use of the 


Krasovskii-LaSalle Invariance Principle. Another situation is when the target sets in P2 are A; = {0} and 


J yl 
1y@ ve) x Zz ji Zz y2 Wy 
D'@I |> | > De! 
J ya 
A, 
ut =. 
I< 
Hy 
A 
P-1y@v%X(t) ee 
A 
1@@()'@I nf 1@@O(H@I 


FIGURE 76.7 Block diagram for the adaptive scheme (Equations 76.21, 76.39, and 76.40) for position coordination. 
Aj; must be interpreted as zero for the leader. 0 denotes the concatenation of the vectors 0; — 8. The adaptive module 
in the feedback path preserves the passivity properties of the closed-loop system. 
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FIGURE 76.8 Left: Snapshots of the formation in the adaptive design with constant reference velocity v4. The 
adaptation is turned on at t = 10, after which point the trajectories converge to the desired formation. Right: The 
relative distance variables z;, plotted as a function of time. 


the regressor vector P(t) is persistently exciting, in which case parameter convergence is established with 
an application of the Teel-Matrosov Theorem [26]. 

As an illustration of the adaptive redesign, we now revisit the example (Equation 76.24) and suppose 
that v4(t) is available only to agent 1. We modify the feedback law (Equation 76.25) for the agents i = 2,3 
as 

uj = —Kj(x; 9) +3, +u™, K;=K} >0, (76.41) 


where the signal 9; and its derivative }; are available for implementation from the parametrization 
(Equation 76.39) and the update law (Equation 76.40). In the simulation presented in Figure 76.8, we take 
the constant reference velocity v4 = [0.2 0.2] and start with the adaptation turned off. Since agents 2, 3 
possess incorrect information about v4 and since there is no adaptation, the relative distances z, do not 
converge to their prescribed sets Ay = {zx : |z,| = 1}. At t= 10, we turn on the adaptation for agents 2 
and 3, which results in convergence to the desired distances |z;,| = 1 asymptotically. 

As a case study for the adaptive redesign, [27] investigated a gradient climbing problem in which the 
leader performs extremum seeking to reach the minima or maxima of a field distribution and the other 
agents maintain a formation with respect to the leader. To incorporate an extremum seeking algorithm 
in the motion, [27] let the reference velocity v4(t) be determined autonomously by the leader, in the form 
of segments vé(t), t € [t,, th41], that are updated in every iteration according to the next Newton step. To 
calculate this Newton step, the leader performs a dither motion from which it collects samples of the field 
and generates finite-difference approximations for the gradient and the Hessian. The adaptive redesign 
discussed above treats the Newton direction as the unknown parameter vector in Equation 76.38 and 
allows the followers to estimate this direction. This case study raised several new problems which led to 
a refinement of the basic adaptive procedure. Among these problems is a judicious tuning of the design 
parameters to ensure that the followers respond to the Newton motion while filtering out the dither 
component. 


76.4 Passivity Approach to Biochemical Reaction Networks 


76.4.1 The Secant Criterion for Cyclic Networks 


Reference [28] proposed a passivity-based analysis technique for cyclic biochemical reaction networks, 
where the end product of a sequence of reactions inhibits the first reaction upstream. This technique 
recovered the secant criterion [7,8] developed earlier by mathematical biologists for the local stability of 
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FIGURE 76.9 Cyclic feedback interconnection of dynamic blocks Hj,..., Hn. 


such reactions and strengthened it to become a global stability test. Cyclic reaction networks are of great 
interest because, as surveyed in [29], they are widespread in gene regulation, cell signaling, and metabolic 
pathways. Unlike positive feedback systems which constitute a subclass of monotone systems [30], the 
negative feedback due to inhibition gives rise to the possibility of attractive periodic orbits. Indeed, a 
Poincaré-Bendixson Theorem proven in [31] for negative feedback cyclic systems of arbitrary order 
shows that bounded trajectories converge either to fixed points or to periodic orbits. Stability criteria for 
cyclic networks are thus important for determining which parameter regimes guarantee convergence to 
fixed points and which regimes yield oscillations. 

To evaluate local stability properties of negative feedback cyclic systems, [7,8] analyzed the Jacobian 
linearization, represented in Figure 76.9 as the feedback interconnection of first-order linear blocks 
Hi(s) = yi/(tis + 1). They then proved that the interconnected system is Hurwitz if the dc gains y; satisfy 
the secant criterion: 

Vier Vn < sec(/n)”. (76.42) 


In contrast to a small-gain condition which would restrict the right-hand side of Equation 76.42 to be 1, 
the secant criterion exploits the phase properties of the feedback loop and allows the gain to be arbitrarily 
large when n = 2, and to be as high as 8 when n = 3. 

Local stability of the equilibrium proven in [7,8], however, does not rule out the possibility of periodic 
orbits as shown in [29] with the example: 


xy = xy + (x3), 2 = —X2 +X], 3 = —X3 +X, (76.43) 


where the nonlinearity (x3) = e1063—-D 4 9. 1sat(25(x3 — 1)) hasa negative slope of magnitude y3 = 7.5 


at the unique equilibrium x; = x2 = x3 = 1. With y; = y2 = 1, the local secant criterion (Equation 76.42) 
guarantees asymptotic stability of the equilibrium as in Figure 76.10a. However, the numerical simulation 
in Figure 76.10b indicates that an attractive periodic orbit exists in addition. 

To develop a global stability test for cyclic networks, [28] exploited a passivity property that is implicit 
in the local secant criterion [32]. To make this property explicit, the first step in [28] is to break down 
the network into n nonlinear subsystems representing the dynamics of each species, interconnected 
according to the cyclic structure in Figure 76.5. The second step is to verify, for each block Hj, the output 
strict passivity (OSP) property [33,34]: 


Si < = + YViUiyis (76.44) 


where u; and y; denote the input and the output of Hj, and S;(x; — x;‘) is a positive definite storage function 
of the deviation of the concentration x; from its equilibrium value x. The third step is to construct a 
Lyapunov function for the network from a weighted sum of these storage functions S; and to prove that a 
set of weights that render its derivative negative definite exists if and only if the secant criterion (Equation 
76.42) holds. In this new procedure, the first-order blocks Hj(s) = yi/(tis + 1) employed in the local 
secant criterion are replaced by nonlinear passive systems with OSP gain y; as in Equation 76.44 and the 
secant condition (Equation 76.42) guarantees global asymptotic stability for the network. 

How does one verify the OSP property (Equation 76.44) and estimate the gain y;? Although this 
task may appear intractable for highly uncertain biological models, [28,35] gave procedures to verify OSP 
without explicit knowledge of the nonlinearities and the equilibrium value x;*. Instead, one verifies OSP by 
qualitative arguments that exploit the monotone increasing or decreasing properties of the nonlinearities, 
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FIGURE 76.10 The trajectories of Equation 76.43 starting from initial conditions (a) x =[1.1 1.1 1], and (b) 
x = [1.2 1.2 1.2], projected onto the x; — x2 plane. The trajectory in (a) converges to the equilibrium x} = x2 = x3 = 1, 
while the trajectory in (b) converges to a periodic orbit. 


such as Michaelis-Menten and Hill equations that arise in activation and inhibition models in enzyme 
kinetics [36]. Once OSP is verified, an upper bound on the gain y; is obtained by inspecting the maximum 
slope of the steady-state characteristic curve y; = k;(uj), where y; denotes the steady-state response of the 
output y; to a constant input uj; = uj. 

As an illustration of the global secant test, consider the following simplified model of a mitogen 
activated protein kinase (MAPK) cascade with inhibitory feedback, proposed in [37,38]: 


b dq — 
ee 1X] 11 — x1) we (76.45) 
cy +x) ey +(1—x)) 1+kx3 
box: ay(1—x 
So a(1 = x2) xy (76.46) 
C2+x2 e2+(1—x2) 
b d3(1 — 
en ee 3(1— x3) x, (76.47) 
c3+x3  €3+(1—x3) 


where the variables x; € [0, 1] denote the concentrations of the active forms of the proteins, and the terms 
1 — x; correspond to the inactive forms (after an appropriate nondimensionalization that scales the sum of 
the active and inactive concentrations to 1). The inhibition of x; by x3 in this model is due to the decreasing 
function 1 /(1 + kx3) in Equation 76.45, the steepness of which is determined by the parameter k. With 
the coefficients b} = ey = cy = 2 = 0.1, 9 =eQ = C3 = €3 = 0.01, 63 = 0.5, dy = dp = d3 = 1, wp = 0.3, 
we obtained the OSP gains y; numerically for various values of k. This numerical experiment showed 
that the secant condition y;y2y3 < 8 is satisfied in the range k < 4.35, which reduces the gap between the 
small-gain estimate k < 3.9 given in [39] and the Hopf bifurcation value k = 5.1. The secant test further 
guarantees global asymptotic stability which cannot be ascertained from a bifurcation analysis. 


76.4.2 Generalization to Other Network Topologies 


The passivity-based analysis outlined above makes the key property behind the local secant criterion 
explicit, extends it to be a global stability test, and further opens the door to a generalization of this test to 
network topologies other than the cyclic structure. Reference [35] achieved this generality by representing 
the reaction network with a directed graph and by making use of the concept of diagonal stability [9,10] 
to expand the passivity-based analysis to such a graph. The nodes x1,...,x, in this directed graph denote 
the concentrations of the species and the links k = 1,..., m represent the reactions, as in Figure 76.11. In 
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particular, solid links represent activation terms which play the role of positive feedback and dashed links 
represent inhibitory terms which act as negative feedback. 

To derive a stability test that mimics the secant criterion, [35] breaks down the network into m 
subsystems each associated with a link and forms an m x m dissipativity matrix E of the form 


-1/y¥1 ifk=1 
ek = 4 sign(link k) if source(/) = sink(k) (76.48) 
0 otherwise, 


where y; is the OSP gain for /th subsystem as in Equation 76.44 and the sign of a link is +1 or —1 
depending on whether it represents positive or negative feedback. This matrix incorporates information 
about the passivity properties of the subsystems, the interconnection structure of the network, and the 
signs of the interconnection terms. 

To determine the stability of the network, [35] checks the diagonal stability of E; that is, the existence 
of a diagonal solution P > 0 to the Lyapunov equation E’ P + PE < 0. When such a P exists, its diagonal 
entries serve as the weights of the storage functions in a composite Lyapunov function. It is important 
to note that here diagonal stability is not used as a local stability test, but is employed to construct a 
composite Lyapunov function as in the large-scale systems literature [2,40,41]. By taking into account 
the signs of the off-diagonal terms in Equation 76.48, however, the diagonal stability test exploits the 
phase properties of the feedback loops in the network and avoids the conservatism of small-gain-type 
dominance approaches prevalent in large-scale systems studies. 

This diagonal stability test encompasses the secant criterion because, as shown in [28], when E is 
constructed according to the cyclic interconnection topology, its diagonal stability is equivalent to the 
secant condition (Equation 76.42). For other practically important network structures, [35] obtained 
variants of the secant criterion by investigating when the dissipativity matrix E is diagonally stable. As an 
illustration, for the feedback configurations (a) and (b) in Figure 76.11, the dissipativity matrices obtained 
according to Equation 76.48 are 


1 1 
-— 0 0 —1 —— 0 0 1 
V1 
1 e 1 
1 -— -1 0 1 -— -1 0 
B= Y2 j : Ey = Y2 1 (76.49) 
0 1 -— 0 0 1 -— 0 
¥3 Y3 
1 1 
0 1 0 -— 0 1 0 -— 
V4 v4 


As shown in [35], matrix E, is diagonally stable if y; y2y4 < 8, and Ey is diagonally stable if y: y2y4 < 1. 
For the feedback configuration in Figure 76.11c, the dissipativity matrix is 


1 
— 0 0 1 0 
V1 
1 
1 —— -l 0 0 
y2 : 
E.= 0 1 —— 0 1 (76.50) 
¥3 
1 
0 1 0 — 1 
V4 
1 
0 0 0 1 -— 
¥5 


and a necessary condition for its diagonal stability is 


yiy2V4+ yays <1. (76.51) 
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FIGURE 76.11 Three feedback configurations proposed in [42] for MAPK networks in PC-12 cells. The nodes x1, 
x2, and x3 represent Raf-1, Mek1/2, and Erk1/2, respectively. The dashed links indicate negative feedback signals. 
Depending on whether the cells are activated with (a) epidermal or (b) neuronal growth factors, the feedback from 
Erk1/2 to Raf-1 changes sign. (c) An increased connectivity from Raf-1 to Erk1/2 is noted in [42] when neuronal 
growth factor activation is observed over a longer period. 


Although the necessary condition (Equation 76.51) does not depend on y3, a numerical investigation 
shows that, unlike the feedback configurations (a) and (b), y3 is implicated in the diagonal stability for 
the configuration in Figure 76.11c. 


76.4.3 Passivity as a Certificate of Robustness to Diffusion 


Thus far we assumed a well-mixed reaction environment and represented the concentration of each 
species i= 1,...,n with a lumped variable x;. The study of spatially distributed reaction models is of 
interest because the presence of diffusion in the spatial domain can lead to subtle instability mechanisms 
in an otherwise stable reaction system [43]. In contrast, the passivity-based stability tests presented 
in Sections 76.4.1 and 76.4.2 rule out such mechanisms and guarantee robustness against diffusion. 
References [29,35] studied both reaction-diffusion PDE models and compartmental ODE models where 
the compartments represent a discrete set of spatial domains as further described below, and proved that 
the secant condition and its generalization in Section 76.4.2 above guarantee global asymptotic stability 
of the spatially homogeneous equilibrium. This homogeneous behavior is illustrated in Figure 76.12 on 
the MAPK example (Equations 76.45 through 76.47) where the concentrations x;(t, £) are now functions 
of the spatial variable &, and the dynamic equation for each i = 1, 2,3 is augmented with diffusion terms. 

The structural property that ensures robustness against diffusion is particularly transparent in a com- 
partmental ODE model in which the state vector X/ incorporates the concentrations be of species 
i=1,...,n in compartment j= 1,...,N. To make the structure of this Nnth order ODE explicit we 
introduce a new, undirected, graph G in which the nodes represent the compartments and the links 
describe the interconnection of the compartments. Denoting by Lg the Laplacian matrix for this graph, 
we obtain the block diagram in Figure 76.13, where the feedforward blocks %; are copies of the lumped 


xy y(t, &) 23 (t, 6) 


FIGURE 76.12 Solutions of the MAPK model (Equations 76.45 through 76.47) augmented with diffusion terms: & 
represents the spatial coordinate and the solutions converge to the spatially homogeneous equilibrium. 
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FIGURE 76.13 Diffusive coupling between the compartments ©; in Equation 76.52. Lg is the Laplacian matrix for 
the graph G representing the interconnection of the compartments, and the concatenated vectors u and y denote 
u=[uf---uy]! and y=[yf ---yxl" 


reaction model perturbed by the diffusion input uj: 
YM =RK)+uyj, yj = CX, uj yj, Xi ER", (76.52) 


and C is a diagonal matrix whose entries are the diffusion coefficients of the species. 

Stability of the interconnection in Figure 76.13 then follows from the results of [44] on positive operators 
with repeated monotone nonlinearities, extended to multivariable nonlinearities in [45]. If the decoupled 
system (Equation 76.52) with uj = 0 admits a Lyapunov function V(X’), then, for the coupled system, the 
sum of these Lyapunov functions for each compartment satisfies 


x! 
N N 
‘ Yo Ve) = SO VV) EF) -[VV(X1)- VV(XN)Lg @C)} 5], (76.53) 
j=l j=l XN 


where the first term on the right-hand side is negative definite. The second term is due to the coupling of 
the compartments and includes the repeated nonlinearity V V(-). Because the graph Laplacian matrix Lg 
is doubly hyperdominant with zero excess, it follows from [44,45] that the second term on the right-hand 
side of Equation 76.53 is nonpositive if VV(C~!-) is a monotone mapping as defined in [45]. Indeed, 
under mild additional assumptions, the Lyapunov functions V(X’) constructed in [28,29,35] consist of a 
sum of convex storage functions of x; which guarantee the desired monotonicity property, thus proving 
negative definiteness of Equation 76.53. 


76.5 Conclusions and Future Topics 


The notion of passivity emerged from energy conservation and dissipation concepts in electrical and 
mechanical systems [11] and became a fundamental tool for nonlinear system design and analysis [33,34]. 
In this chapter we showed that passivity is a powerful design and analysis approach for several types of 
networks. We have further identified recurrent interconnection structures in these networks, such as the 
symmetric-coupling structure in Figures 76.2, 76.3, 76.5 through 76.7 and 76.13 and the cyclic structure 
in Figure 76.9, which are well suited to this passivity approach. 

Verification and assignment of passivity properties were hampered by the unavailability of the network 
equilibrium to the components in the communication and biological network examples in Sections 76.2 
and 76.4. This difficulty was overcome by exploiting monotone increasing or decreasing properties of 
nonlinearities which led to incremental forms of passivity that do not depend on the equilibrium location. 
Likewise, the unavailability of the network reference velocity in the motion coordination study of Section 
76.3 was overcome with an adaptive redesign. Despite these encouraging results, further studies are 
needed for achieving passivity of the components when global network parameters are unavailable. 
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